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FOREWORD 


On  July  20,  1981,  EPA  requested  NBS  to  review  the  analysis  for  organic  chemicals 
performed  by  EPA  in  its  Love  Canal  Monitoring  Study.  In  response  to  that 
request,  this  review  has  been  prepared  by  a panel  of  NBS  scientists  with 
expertise  in  organic  analytical  chemistry,  quality  assurance  and  statistics. 

The  review  is  based  upon  written  material,  provided  by  EPA  to  NBS,  related  to 
the  acquisition  of  organic  analytical  data  for  samples  collected  from  the  Love 
Canal  vicinity  by  EPA  contractors. 

The  charge  from  EPA,  accepted  by  NBS  on  August  17,  1981,  was  to  carry  out  the 
following  seven  tasks: 

1.  Review  the  sample  collection  and  analytical  protocols  for  organic 
chemicals  in  air,  water,  soils,  sediments  and  biota  to  determine 'their 
appropriateness  for  identifying  and  measuring  the  substances  of  interest 
to  EPA  at  Love  Canal. 

2.  Review  the  adequacy  of  the  quality  assuranc e/qual ity  control  protocols 
for  all  of  the  media  in  Task  1 to  assure  the  validity  of  substance 
identification  and  analytical  measurements. 

'3.  Review  the  results  of  the  quality  assurance  program  presented  in  the  EPA 
Love  Canal  report  to  assess  the  performance  of  the  analytical  program. 

4.  Review  as  necessary  any  of  the  reports  of  the  on-site  laboratory  audits 
carried  out  by  EPA. 

5.  Review  the  results  of  the  EPA  audit  of  GC-MS  tapes  to  evaluate  the 
quality  of  the  analytical  program. 

6.  Review  any  of  the  monitoring  data  as  necessary  to  help  in  the  overall 
review  of  the  program. 

7.  Prepare  a report  on  the  overall  adequacy  of  the  EPA  analytical  and 
quality  assurance  protocols  to  meet  the  organic  chemical  monitoring 
goals  (accuracy  and  precision)  as  set  forth  in  the  EPA  Love  Canal 
Monitoring  PI  an. 

On  August  18,  1981,  NBS  received  an  initial  set  of  dociments  for  review,  and  on 
August  25,  NBS  was  given  an  oral  briefing  by  officials  of  EPA  and 
representatives  of  the  prime  contractor,  the  GCA  Corporation,  on  the  conduct  of 
the  Love  Canal  Monitoring  Study.  At  the  time  NBS  initiated  its  review,  no  final 
report  on  the  Love  Canal  Monitoring  Study  was  available.  As  a result  of  the 
unavailability  of  such  a report.  Tasks  3 and  7 were  modified  to: 

3.  Review  the  results  of  the  quality  assurance  program  as  revealed  in 
documents  provided  to  NBS  describing  the  performance  of  the  analytical 
program. 
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and 


7.  Based  upon  written  material  provided  by  EPA,  prepare  a report 
summarizing  the  reviews  conducted  by  NBS  in  Tasks  1 to  6,  paying 
particular  attention  to  conclusions  drawn  by  EPA  concerning  the 
precision  and  accuracy  of  organic  analytical  data  from  the  Love  Canal 
Monitoring  Study. 

In  order  to  give  structure  and  specificity  to  the  review,  the  NBS  panel  devised 
a set  of  questions  to  serve  as  guidance  for  the  evaluation  of  the  organic 
analytical  portion  of  the  EPA  Love  Canal  Monitoring  Study.  This  set  of 
questions,  designated  in  correspondence  between  EPA  and  NBS  as  the  template  for 
the  review,  was  transmitted  to  EPA  on  October  13. 

On  December  9,  1981,  NBS  submitted  its  review  of  material  received  from  EPA. 

That  material  consisted  of  a variety  of  documents  including  contractor  plans, 
tables  of  data  derived  from  samples  taken  from  the  Love  Canal  vicinity,  tables 
of  data  derived  from  internal  and  external  performance  check  samples,  quality 
assurance  audit  reports,  miscellaneous  letters  and  reports,  and  a preliminary 
draft  of  portions  of  a final  report.  This  latter  document  was  designated  as 
OEM-LC-2  by  NBS.  The  December  9 review  identified  a number  of  deficiencies  in 
the  Love  Canal  Monitoring  Study  on  the  basis  of  the  written  material  provided  by 
EPA.  In  the  cover  letter  to  EPA  accompanying  the  December  9 review,  NBS  stated 
that  the  responses  to  the  template  questions  would  be  submitted  to  EPA  in 
January  1982. 

On  December  15,  EPA  requested  NBS  to  review  additional  material,  and  on 
December  17,  EPA  provided  NBS  with  a copy,  designated  #9,  of  a draft  report. 

NBS  was  informed  that  those  sections  of  the  draft  report  which  dealt  with  the 
quality  assurance  aspects  of  the  study  were  still  in  the  process  of  being 
revised.  NBS  was  requested  to  work  with  the  revised  versions.  Since  the 
additional  material  influenced  the  NBS  responses  to  the  template  questions,  NBS 
decided  not  to  transmit  the  responses  until  the  additional  material  had  been 
reviewed.  By  February  17,  1982,  the  last  of  the  additional  material  provided  by 
EPA,  which  included  revisions  of  the  draft  report  and  a copy  of  the  final  report 
of  the  prime  contractor,  was  received.  At  the  request  of  EPA,  NBS  briefed  EPA 
on  the  contents  of  this  review  on  April  15,  1982. 

The  present  Review  covers  all  material  received,  with  the  exception  of  the 
aforementioned  OEM-LC-2,  which  was  replaced  by  the  draft  report.  All  documents 
received  by  NBS  are  catalogued  at  the  end  of  this  Review. 

Finally,  the  authors  of  this  Review  wish  to  acknowledge  the  advice, and 
assistance  of  Prof.  Michael  Gross,  Mr.  Hugh  Huffman,  and  Dr.  James  Sphon. 
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INTRODUCTION 


This  is  a review  by  the  National  Bureau  of  Standards  of  the  analysis  for  organic 
chemicals  conducted  by  the  U.  S.  Environmental  Protection  Agency  in  the  Love 
Canal  Area  of  Niagara  Falls,  New  York.  In  conducting  this  review,  NBS  has  been 
guided  by  the  seven  tasks  specified  by  EPA  in  the  charge  given  to  NBS. 

To  give  additional  structure  to  the  review  and  to  ensure  that  all  appropriate 
aspects  of  the  Organic  Analytical  Program  of  the  Love  Canal  Monitoring  Study 
were  considered,  NBS  devised  a series  of  questions  concerning  various  aspects  of 
the  program  to  serve  as  a detailed  guide  for  the  review.  This  set  of  questions 
has  been  designated  in  correspondence  between  NBS  and  EPA  as  the  "template"  for 
the  review  and  the  responses  to  the  template  questions  are  included  as  Appendix 
A of  this  Revi ew. 

In  this  Review,  NBS  has  commented  upon  deficiencies  or  limitations  that  NBS  has 
identified  in  the  Love  Canal  Monitoring  Study.  Not  all  of  these  deficiencies  or 
limitations  influence  the  conclusions  drawn  by  EPA  in  its  draft  report  and  some 
have  already  been  described  and  interpreted  by  EPA.  The  purpose  of  including 
such  comments  is  to  call  to  the  attention  of  readers  particular  limitations 
which  might  not  be  immediately  apparent  or  which  should  be  considered  by  those 
who  wish  to  use  the  EPA  data  to  draw  independent  conclusions.  When  deficiencies 
or  limitations  are  identified  which  might  affect  the  conclusions  drawn  by  EPA  in 
its  draft  report,  they  are  so  noted. 

Finally,  since  the  NBS  review  was  to  be  concerned  solely  with  the  analysis  for 
organic  chemicals,  other  aspects  of  EPA's  Love  Canal  Monitoring  Study  have  been 
excluded,  by  agreement,  from  this  review.  Specifically,  these  are: 

1.  The  choice  of  sample  locations  and  media  sampled.  NBS  did  review, 
however,  how  the  sample  collection  protocols  might  have  affected  the 
chemical  integrity  of  the  samples  being  collected. 

2.  The  reduction  of  acquired  data  leading  to  contamination  level  simmaries 
and  distribution  patterns.  NBS  did  review,  however,  those  statistical 
procedures  associated  with  the  determination  of  precision,  accuracy, 
and  limits  of  detection  in  the  analysis  of  environmental  samples  for 
organic  chemicals. 

3.  The  choice  of  compounds  which  would  be  determined  in  the  samples 
collected  from  the  Love  Canal  vicinity.  NBS  did  consider,  however,  the 
effect  of  selecting  a list  of  targeted  substances  on  the  overall 
analytical  program. 

4.  The  analytical  programs  for  inorganic  chemicals  and  radioactivity . 

5.  The  conclusions  drawn  by  EPA  from  the  Love  Canal  Monitoring  Study.  NBS 
did  consider,  however,  whether  or  not  the  quality  and  reliability  of 
the  organic  analytical  data  might  affect  the  conclusions. 
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REVIEW  OF  THE  ANALYSIS  FOR  ORGANIC  CHEMICALS 
IN  THE  EPA  LOVE  CANAL  MONITORING  STUDY 


EPA  STUDY  GOALS 


The  EPA  Love  Canal  Monitoring  Study  was  undertaken  to  accomplish  three 
expl ici t goals: 

1.  Determine  the  current  extent  and  degree  of  chemical  contamination 
in  the  area  defined  by  the  emergency  declaration  order. 

2.  Assess  the  near-term  and  long-term  implications  of  groundwater 
contamination  in  the  general  vicinity  of  Love  Canal. 

3.  Provide  an  assessment  (from  an  environmental  contamination 
perspective)  of  the  habitability  of  residences  included  in  the 
emergency  declaration  order. 


EPA  STUDY  DESIGN  FOR  THE  ANALYSIS  FOR  ORGANIC  CHEMICALS 


The  monitoring  effort  involved  the  analysis  of  environmental  samples  for 
more  than  100  targeted  substances.  The  substances  targeted  for  analysis 
were  selected  from  those  known  to  have  been  buried  in  Love  Canal,  those 
previously  observed  in  the  environment  near  Love  Canal,  and  the  EPA  list  of 
priority  pollutants.  In  addition  to  the  targeted  substances,  the  EPA  draft 
report  cites  as  one  of  the  major  safeguards  in  the  monitoring  study  the 
identification  of  the  20  most  abundant  non-targeted  substances  in  each 
environmental  sample. 

Four  media — air,  water,  soil  and  sediments,  and  biota--were  to  be 
independently  studied  using  gas  chromatography-mass  spectrometry  (GC-MS)  as 
the  primary  analytical  technique.  A detailed  quality  assurance  program  was 
designed  for  which  the  number  of  quality  assurance  samples  to  be  analyzed 
equaled  the  number  of  field  samples.  From  the  analytical  data,  EPA  was  to- 
generate  a validated  data  base  for  the  environmental  samples. 


SUMMARY  OF  THE  NBS  REVIEW 


In  response  to  a request  from  EPA,  NBS  has  reviewed  the  analysis  for 
organic  chemicals  in  the  EPA  Love  Canal  Monitoring  Study.  The  review  is 
based  upon  written  documentation  provided  to  NBS  during  the  period 
August  18,  1981  to  February  17,  1982.  (See  Appendix  B for  the  list  of 
documents  received.)  The  review  was  confined  to  those  subjects  discussed 
in  the  Foreword  and  the  Introduction  to  this  report. 
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• EPA's  stated  goals  and  objectives,  as  they  applied  to  the  analysis  for 
organic  chemicals,  were  appropriate  in  that  media,  substances,  potential 
patterns  of  contamination,  and  temporal  variation  were  considered.  They 
were  not,  however,  quantitatively  explicit.  EPA  did  not  define  at  the 
outset  of  the  study  the  limits  of  detection  or  the  precision  and  accuracy 
required  to  permit  comparison  of  levels  of  contamination  in  the  Declaration 
Area  with  levels  which  constitute  an  environmental  hazard  or  with  levels 
currently  found  in  U.S.  cities. 

In  the  implementation  of  the  monitoring  program,  the  requirement  for  the 
identification  of  the  20  most  abundant  non-targeted  substances  was 
generally  not  met.  Consequently,  the  possibility  of  contamination  by  non- 
targeted  substances  has  not  been  conclusively  excluded.  This  deficiency 
does  not  invalidate,  however,  the  investigation  of  targeted  substances. 

The  quantitation  limits  realized  by  the  water  monitoring  laboratories  for 
individual  analytes  varied  by  more  than  a factor  of  ten.  Few  laboratories 
reported  quantitation  limits  as  good  as  those  reported  by  the  EPA 
laboratory  responsible  for  the  water  monitoring  effort.  Comparisons  of 
quantitation  limits  with  data  provided  by  EPA  from  other  studies  indicated 
that  the  contract  laboratories  were  not  uniformly  performing  at  the  state 
of  the  art  in  the  water  monitoring  program. 

In  any  study  involving  measurements,  conclusions  are  bounded  by  measurement 
error  and  limits  of  detection.  EPA  has  not  fully  used  its  quality 
assurance  data  to  derive  estimates  of  precision,  accuracy,  and  limits  of 
detection.  EPA  has  not  incorporated  estimates  of  precision,  accuracy,  and 
limits  of  detection  into  its  validated  data  base.  Unless  measured  values, 
including  "none  detected,"  are  accompanied  by  estimates  of  uncertainty, 
they  are  incomplete  and  of  limited  usefulness  for  further  interpretation 
and  for  drawing  conclusions. 


SPECIFIC  FINDINGS 


In  the  remainder  of  this  Review,  specific  comments  are  grouped  according  to 
the  seven  tasks  in  the  charge  to  NBS  from  EPA.  Because  of  the  variations 
in  the  approach  taken  in  the  analysis  of  samples  from  each  of  the  four 
different  media,  these  comments  often  pertain  to  only  one  of  the  four 
media.  Additional  detailed  comments  can  be  found  in  the  answers  to  the 
template  questions  appearing  in  Appendix  A. 


1.  "Review  the  sample  collection  and  analytical  protocols  for  organic 

chemicals  in  air,  water,  soils,  sediments  and  biota  to  determine  their 
appropriateness  for  identifying  and  measuring  the  substances  of 
interest  to  EPA  at  Love  Canal." 


The  approach  of  identifying  a list  of  targeted  substances  and 
requiring  a search  for  the  20  most  abundant  non-targeted  substances 
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was  sound.  Targeted  substances  were  chosen  from  appropriate  lists  and 
from  the  results  of  earlier  Love  Canal  investigations.  The  criteria 
established  for  the  selection  of  targeted  substances  were  reasonable. 
However,  EPA  has  not  explicitly  related  included  or  excluded 
substances  to  specific  criteria.  The  substances  sought  were 
representative  of  broad  classes  of  organic  chemicals. 

Chemical  identification  and  quantitation  by  gas  chromatography-mass 
spectrometry  (GC-MS)  was  appropriate  to  the  general  goals  and 
objectives  of  the  study  and  represented  the  best  overall  technique  for 
monitoring  organic  chemicals  in  environmental  samples.  NBS  is  not 
aware  of  any  fundamentally  different  analytical  techniques  for  the 
identification  and  quantitation  of  organic  compounds  which  would  have 
been  more  suitable  for  the  study  as  planned.  For  some  individual 
substances,  alternative  analytical  methods  might  have  been  more 
suitable  (for  example,  the  use  of  a collection  medium  other  than  TENAX 
for  benzene  and  toluene--a  problem  recognized  in  EPA's  draft  report). 

The  sample  collection  and  analytical  protocols  were  generally  complete 
for  the  air  and  water  monitoring  efforts  and  have  been,  to  varying 
degrees,  widely  used  in  environmental  monitoring  for  several  years. 

The  protocols  for  soil  and  sediments  and  for  biota  were  modifications 
of  the  water  methods  and  were  to  be  evaluated  during  the  course  of  the 
study.  (The  biota  monitoring  effort  was  experimental  in  nature  and 
the  data  resulting  from  this  effort  were  not  used  by  EPA  to  draw 
conclusions  from  its  study.)  Some  ambiguity  existed  in  the  details  of 
the  protocols  which  could  have  affected  the  quality  of  the  analytical 
data. 

The  monitoring  program  design  did  not  anticipate  the  possibility  that 
a significant  fraction  of  the  samples  might  not  contain  any  detectable 
levels  of  analytes.  Consequently,  the  significance  of  the  limits  of 
detection  and  quantitation  and  how  these  limits  might  affect  the 
conclusions  drawn  from  the  study  were  not  adequately  considered. 


2.  "Review  the  adequacy  of  the  quality  assurance/quality  control 

protocols  for  all  of  the  media  in  Task  1 to  assure  the  validity  of 
substance  identification  and  analytical  measurements." 


The  number,  nature,  and  frequency  of  analysis  of  quality  assurance 
samples  specified  by  EPA  should  have  been  adequate  to  maintain  quality 
control. 

The  quality  assurance  plans  for  the  four  media  (air,  water,  soil  and 
sediments,  and  biota)  were  similar  in  scope  but  varied  in  detail.  The 
quality  assurance  plans  for  the  participating  laboratories  were,  in 
general,  complete  and  followed  the  outline  given  in  the  overall 
Quality  Assurance  Plan.  These  plans  did,  however,  vary  in  detail  from 
one  laboratory  to  another.  The  variations  in  detail  could  lead  to 
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difficulties  in  comparing  data  obtained  from  different  laboratories 
and  different  medi a. 

The  Quality  Assurance  Plan  encouraged  but  did  not  require  the  use  of 
control  charts.  Uniform  reporting  of  corrective  actions  taken 
when  quality  control  was  lost  was  not  required.  It  will  be  difficult 
at  best  to  evaluate  further  any  data  reported  by  a laboratory  without 
an  accompanying  quality  control  chart  (or  equivalent  data). 

The  Quality  Assurance  Plan  did  not  address  how  the  quality  assurance 
data  would  be  used  to  qualify  the  environmental  data,  that  is,  to 
assign  uncertainty  limits  to  each  measured  value,  including  "none 
detected." 

Estimates  of  the  limits  of  detection  are  critical  to  the  conclusions 
which  can  be  drawn  when  the  majority  of  samples  contain  no  measurable 
concentration  of  analytes.  The  Quality  Assurance  Plan  did  not 
anticipate  such  a possibility. 


3.  "Review  the  results  of  the  quality  assurance  program  as  revealed  in 
documents  provided  to  NBS  describing  the  performance  of  the  analytical 
program." 


The  quantitation  limits  realized  by  the  water  monitoring  laboratories 
for  individual  analytes  varied  by  more  than  a factor  of  ten.  Few 
laboratories  reported  quantitation  limits  as  good  as  those  reported  by 
the  EPA  laboratory  responsible  for  the  water  monitoring  effort. 
Comparisons  of  quantitation  limits  with  data  from  other  studies  (where 
available)  indicated  the  laboratories  were  not  uniformly  performing  at 
the  state  of  the  art. 

The  limits  of  quantitation  to  be  achieved  by  the  air  monitoring 
laboratories  were  set  at  concentration  levels  comparable  to  levels 
observed  for  some  U.S.  cities. 

As  demonstrated  by  the  audit  of  the  GC-MS  computer  records,  the  water 
and  the  soil  and  sediments  monitoring  laboratories  did  not  generally 
identify  non-targeted  substances.  This  aspect  of  the  performance  of 
the  air  monitoring  laboratories  was  not  evaluated  by  EPA. 

In  isolated  instances,  contamination  of  the  samples  during  sample 
collection  or  analysis  may  have  obscured  possible  trends  and  patterns 
in  the  environmental  data. 

The  transcription  of  the  data  and  the  verification  of  the  accuracy  of 
the  transcription  and  transmittal  of  the  data  into  the  final  data  base 
were  adequately  demonstrated. 

The  documentation  provided  by  EPA  indicated  that  the  quality  control 
limits  set  in  the  Quality  Assurance  Plan  widened  during  the  study. 
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The  degree  of  change  was  greater  for  the  water  program  than  for  the 
air  program.  The  reasons  for  the  observed  changes  have  not  been 
explained  in  the  documents  provided  by  EPA. 

The  estimation  of  the  precision  and  accuracy  of  the  nunerical  values 
of  concentrations  of  targeted  substances  has  not  been  adequately 
docunented.  Only  a portion  of  the  available  quality  assurance  and 
environmental  data  has  been  used  in  arriving  at  these  estimates. 

The  estimates  are  generally  unconfirmed  and  estimates  for  some  of  the 
analytes  have  not  been  given. 

The  estimation  of  the  limits  of  detection  and  quantitation  are  also 
not  adequately  documented.  Because  the  conclusions  of  the  study  rest 
on  comparisons  of  the  frequency  of  compound  detection,  the  values 
obtained  for  the  limits  of  detection  of  individual  substances  can 
influence  significantly  such  conclusions. 

The  rejection  of  data  has  neither  been  clearly  described  nor 
adequately  justified. 


4.  "Review  as  necessary  any  of  the  reports  of  the  on-site  laboratory 
audits  carried  out  by  EPA." 


The  reports  of  on-site  laboratory  audits  reviewed  by  NBS  described 
audits  conducted  by  EPA  and  by  the  prime  contractor  during  the  early 
stages  of  the  Love  Canal  Monitoring  Study.  These  reports  indicate 
that  all  laboratories  were  having  some  difficulties  and  some 
laboratories  were  having  great  difficulty  at  the  start  of  the  study. 

In  particular,  some  of  the  laboratories  were  experiencing  difficulties 
in  adjusting  to  the  use  of  capillary  colunns. 

The  prime  contractor  maintained  telephone  contact  with  participating 
laboratories  during  the  course  of  the  study,  but  there  is  no  evidence 
of  further  monthly  on-site  visits  as  prescribed  by  the  Quality 
Assurance  Plan.  Such  on-site  audits  would  have  confirmed  whether  or 
not  initial  difficulties  had  been  overcome,  whether  or  not  laboratory 
contamination  of  samples  was  a problem,  whether  or  not  good  laboratory 
practices  were  being  followed,  and  whether  or  not  analog  data  (e.g., 
chromatograms)  from  the  GC-MS  runs  indicated  laboratories  were 
operating  at  the  expected  level  of  performance. 


5.  "Review  the  results  of  the  EPA  audit  of  GC-MS  tapes  to  evaluate  the 
quality  of  the  analytical  program." 


The  EPA  audit  of  GC-MS  computer  records  was  not  an  evaluation  of  the 
quality  of  the  entire  analytical  program,  but  only  an  evaluation  of 
one  aspect  of  that  program,  namely  the  interpretation  of  the  computer 
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records.  The  audit  evaluated  a sample  of  the  computer  records 
generated  in  the  water  and  in  the  soil  and  sediments  monitoring 
programs.  No  audit  of  the  air  or  biota  monitoring  programs  was 
performed. 

The  conclusions  of  the  audit  have  not  been  completely  justified,  e.g., 
the  claim  that  missed  compounds  were  predominantly  in  heavily 
contaminated  samples  is  not  supported. 

The  implications  of  the  audit--that  significant  difficulties  existed 
at  the  limits  of  detection--to  the  conclusions  of  the  study  are  not 
discussed  in  documents  provided  by  EPA. 

Eighty  samples  (water,  soil  and  sediments)  were  audited  for  the 
identification  of  non-targeted  substances.  In  58  of  these  samples, 
both  the  audit  laboratory  and  the  analytical  laboratory  identified  no 
non-targeted  substances.  In  the  remaining  22  samples,  the  audit 
laboratory  identified  84  non-targeted  substances  while  the  analytical 
laboratories  found  only  1.  This  result  indicated  that,  for  the 
samples  audited,  most  laboratories  did  not  identify  non-targeted 
substances. 


"Review  any  of  the  monitoring  data  as  necessary  to  help  in  the  overall 
review  of  the  program." 


Precision,  accuracy,  limits  of  detection,  and  limits  of  quantitation 
have  not  been  incorporated  into  the  validated  data  base.  That  is,  the 
measured  values,  including  "none  detected,"  in  the  validated  data  base 
are  not  accompanied  by  estimates  of  their  uncertainty. 

The  information  given  is  not  sufficient  to  assist  others  in  the 
interpretation  of  the  Love  Canal  data.  Information  on'  1 aboratory-to- 
laboratory  variability  is  incomplete.  In  particular,  recovery  factors 
have  not  been  given  nor  confirmed  for  most  analytes  in  water  and  in 
soil  and  sediments  samples. 

In  the  presentation  of  information,  the  distinction  between  plans  and 
attained  performance  is  not  always  clear.  Examples  selected  to 
support  the  conclusions  of  the  study  have  not  always  been  confirmed  as 
being  truly  typical  of  the  data  in  general. 

Conclusions  of  no  discernible  patterns  or  differences  or  of  no 
significant  c on tami nation  are  bounded  by  the  limits  of  detection  and 
quantitation.  In  the  presentation  of  its  conclusions,  EPA  has  not 
addressed  explicitly  how  those  conclusions  are  influenced  by  the 
limitations  in  its  analytical  data.  Unless  measured  values,  including 
"none  detected,"  are  accompanied  by  estimates  of  uncertainty,  they  are 
incomplete  and  of  limited  usefulness  for  further  interpretation  and 
for  drawing  conclusions. 
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7.  "Based  upon  written  materrial  provided  by  EPA,  prepare  a report 
summarizing  the  reviews  conducted  by  NBS  in  Tasks  1 to  6,  paying 
particular  attention  to  conclusions  drawn  by  EPA  concerning  the 
precision  and  accuracy  of  organic  analytical  data  from  the  Love  Canal 
Monitoring  Study." 


The  foregoing  discussion,  along  with  additional,  detailed  comments 
appearing  in  Appendix  A,  constitute  the  report  summarizing  the  reviews 
conducted  by  NBS  in  Tasks  1 to  6. 


APPENDIX  A 


RESPONSES  TO  THE  TEMPLATE  QUESTIONS 


Template  questions  were  designed  by  NBS  to  serve  as  a guide  for  its  review,  to 
give  structure  to  the  review,  and  to  ensure  that  all  appropriate  aspects  of  the 
analysis  for  organic  chemicals  in  the  Love  Canal  Monitoring  Study  were 
considered.  NBS  has  not  always  answered  each  subquestion  but  has  used  each 
group  of  questions  as  guidance  for  its  review.  All  questions  are  included  for 
completeness. 


I.  Goals  and  Objectives  for  the  Acquisition  of  Organic  Analytical  Data  from  the 
Love  Canal  Monitoring  Study. 

Scope:  To  evaluate  the  consistency  of  the  goals  and  objectives  for  the 

acquisition  of  organic  analytical  data  with  existing  state-of-the-art 
methodology. 


A.  What  were  the  goals  and  objectives  of  the  Organic  Analysis  Project?  Has 
EPA  addressed  in  the  written  documentation  whether  or  not  these  goals 
were  met? 


The  goals  and  objectives  of  the  Love  Canal  Monitoring  Study  were  presented 
on  Page  1 of  the  draft  report  reviewed  by  NBS.  As  stated  therein,  these 
were: 


1.  Determine  the  current  extent  and  degree  of  chemical  contamination  in 
the  area  defined  by  the  emergency  declaration  order. 

2.  Assess  the  near-term  and  long-term  implications  of  groundwater 
contamination  in  the  general  vicinity  of  Love  Canal. 

3.  Provide  an  assessment  (from  an  environmental  contamination 
perspective)  of  the  habitability  of  residences  included  in  the 
emergency  declaration  order. 

These  goals  were  defined  in  somewhat  more  detail  on  Pages  53  and  54  of  the 
draft  report  as  follows: 
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1.  To  characterize  in  each  environmental  mediun  the  incremental  extent 
and  degree  of  chemical  contamination  in  the  Declaration  Area 
directly  attributable  to  Love  Canal. 


2.  To  determine  potential  temporal  variability  in  contamination  and 
infer  the  causal  mechanisms  (for  example,  changes  in  climate) 
influencing  the  observed  contamination  patterns. 


3.  To  determine  if  swales,  sewer  lines,  and  other  geological  features 
(for  example,  sandy  soil  deposits  in  the  form  of  sand  lenses)  had  a 
significant  effect  on  the  migration  of  toxic  substances  from  the 
canal . 

4.  To  determine  the  presence  and  direction  of  ground-water  flow  in  the 
area,  and  evaluate  the  effectiveness  of  the  remedial  construction 
performed  at  Love  Canal. 

5.  To  investigate  the  use  of  locally  available  biological  systems  as 
potential  indicators  of  toxic  substances  present  in  the  environment. 


6.  To  obtain  integrated  multimedia  measurements  of  environmental 
contami nation. 


7.  To  provide  from  an  environmental  perspective  an  assessment  of  the 
relative  habitability  of  Declaration  Area  residences  and  the  short- 
term and  long-term  implications  of  observed  environmental 
contami nation. 


In  the  section  on  implementation  beginning  on  Page  54,  immediately  following 
the  statement  of  the  objectives  of  the  study,  it  is  stated: 

"The  EPA  studies  were  initiated  by  first  identifying  the  data 
requirements  of  the  overall  objectives  and  then  designing  data 
collection  mechanisms  appropriate  for  such  activities." 

These  identified  data  requirements  have  neither  been  described  further  nor 
have  they  been  presented  in  the  report  beyond  the  non-specific  statements  on 
Page  61: 

"The  common  objective  of  these  plans  was  to  collect  and  analyze  a 
statistically  adequate  number  of  samples  to  characterize  accurately 
Declaration  Area  contamination  caused  by  Love  Canal,  and  to  minimize  the 
effects  and  uncertainties  associated  with  the  constrained  .samp  1 ing 
period." 

and  Page  62: 

"...a  primary  goal  of  qualitative  accuracy  for  organic  analyses  (that 
is,  correct  identification  of  detected  substances)  was  established." 
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These  statements  do  not  address  in  quantitative  terms  the  data  requirements 
of  the  goals  and  objectives  of  the  study.  For  example,  the  data 
requirements  could  have  been  made  more  quantitative  by  defining  the 
concentrations  of  targeted  compounds  which  had  to  be  detected  and/or 
quantified  in  order  to  compare  levels  of  contami nation  in  the  Declaration 
Area  with  levels  known  to  constitute  an  environmental  hazard  and/or  with 
levels  currently  found  in  U.S.  cities. 

On  Page  70  of  the  draft  report,  statements  concerning  the  expected 
performance  of  analytical  methodology  are  made: 

"Precisions  better  than  50  percent  RSD  [relative  standard  deviation] 
were  expected  in  water  and  air;  precisions  better  than  100  percent  were 
expected  in  other  media." 

"Furthermore,  it  is  also  known  that  in  water,  the  minimum  method 
quantitation  limits  expected  for  the  methods  used  are  in  the  range  of  1 

to  10  micrograms  per  liter  (parts  per  billion) Quantitation  limits 

below  these  values  were  neither  required  nor  expected  of  the  analytical 
subcontractors,  except  as  noted  previously  for  2,3,7,8-TCDD  and  certain 
pesticides." 

With  the  exception  of  the  2,3,7,8-TCDD,  these  were  expectations  rather  than 
data  requirements.  Without  more  quantitative  specificity,  it  is  difficult 
to  see  how  EPA  could  design  a monitoring  program  to  meet  its  requirements . 

In  the  presentation  of  its  conclusions,  EPA  does  not  address  explicitly  how 
limitations  in  the  analytical  data  affected  the  conclusions.  Whether  or 
not  this  is  a direct  consequence  of  the  absence  of  quantitative  precision 
and  accuracy  goals  cannot  be  determined.  Nevertheless,  those  who  wish  to 
use  the  Love  Canal  data  or  wish  to  interpret  the  conclusions  of  the  Love 
Canal  Monitoring  Study  must  read  and  understand  the  entire  report  in  detail, 
including  the  Appendices,  to  gain  even  a qualitative  sense  of  the 
limitations  of  the  data  gathered  in  the  study. 

The  "primary  goal  of  qualitative  accuracy"  is  partially  addressed  in  the  GC- 
MS  audit  performed  by  EPA  and  is  discussed  in  more  detail  in  Section  VII  of 
this  Appendix. 


B.  Would  the  approach  of  selecting  a list  of  targeted  compounds  in  any  way 
hinder  EPA  from  observing,  identifying,  or  quantifying  significant 
quantities  of  other  compounds?  Should  EPA  review  all  of  the  GC-MS  tapes 
to  determine  if  significant  chemicals  might  have  been  missed? 


The  approach  of  selecting  a list  of  targeted  compounds  should  not  have 
hindered  EPA  from  observing,  identifying,  or  quantifying  other  compounds 
which  are  amenable  to  GC-MS.  Furthermore,  this  approach  is  appropriate  to 
the  study  and  the  list  is  reasonable.  By  restricting  the  analytical 
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methodology  mainly  to  GC-MS,  those  compounds  which  are  either  too  polar  or 
too  nonvolatile  to  analyze  without  derivati zation  would  not  be  detected.  To 
derive  the  list  of  targeted  compounds,  EPA  selected  from  a list  of  chemicals 
which  were  known  to  have  been  dumped  in  the  Love  Canal,  chemicals  found  in 
previous  studies  of  the  Love  Canal,  and  certain  chemicals  generally 
recognized  as  toxic,  associated  with  industrial  production,  or  observed  in 
other  studies  as  environmental  contaminants — the  so-called  "priority 
pollutants".  EPA  should  state  in  its  report  which  compounds  were  selected 
from  which  lists.  The  list  of  targeted  compounds  was  not  all-inclusive. 

For  example,  a previous  study  of  the  Love  Canal  area  (J.  Barkley,  et  al . , 
Biomed.  Mass  Spectrom.  7(4),  139-146  (1980))  cites  estimated  levels  of 
1,1,1-trichloroethane  and  trichloroethylene  in  air  inside  homes  in  the  Love  i 
Canal  area,  but  these  compounds  were  not  in  the  list  of  targeted  air 
compounds  in  Table  1-2,  Page  1-4  of  the  draft  report. 

All  laboratories  were  instructed  to  identify  the  20  most  abundant  substances 
beyond  those  appearing  on  the  target  list.  The  audit  of  the  GC-MS  computer 
records  (discussed  in  Section  VII  below)  indicated  that  the  requirement  for 
identifying  non-targeted  compounds  was  generally  not  met.  Further  review  of 
the  GC-MS  computer  records  for  non-targeted  compounds  missed  in  the  initital 
analyses  would  be  useful  only  if  it  were  to  become  necessary  to  evaluate  the 
significance  of  these  compounds. 

In  the  description  of  the  development  of  the  list  of  targeted  compounds,  it 
was  noted: 

I 

"...the  intentional  inclusion  of  specific  substances  on  the  target  list  i 
that  were  known  to  serve  as  effective  and  efficient  tracers  of 
subsurface  migration  of  leachate  was  designed  to  permit  a comprehensive 
determination  and  assessment  of  migration  patterns  from  the  canal 
source." 

Nowhere  in  the  report  are  these  compounds  identified  or  justified  as 
efficient  tracers,  nor  were  any  results  presented  which  give  the  reader  of 
the  report  an  indication  of  whether  or  not  these  efficient  tracers  migrated 
through  the  subsurface  area  in  the  Love  Canal  region. 


C.  Were  the  accuracy  and  precision  goals  for  the  organic  data  clearly 
established  at  the  outset  and  have  the  criteria  for  these  goals  been 
clearly  expl ained? 


Section  3.3.2,  Pages  68  to  70,  of  the  draft  report  contains  a description  of 
the  precision  and  accuracy  goals  for  the  study.  This  section  deals 
primarily  with  the  accuracy  of  compound  identification  and  provides  some 
references  to  the  precision  which  might  be  expected  for  quantitative 
analyses  such  as  those  conducted  by  EPA  in  its  Love  Canal  study.  Precisions, 
of  2 to  13  percent  relative  standard  deviation  (RSD)  for  water  analyses,  50 
percent  for  air  analyses  and  100  percent  for  other  environmental  media  were 
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quoted  in  the  draft  report  as  generally  being  acceptable.  These  estimates 
of  precision  did  not  serve  as  limits  to  be  used  in  a quality  control 
program,  but  as  guidelines  for  what  could  be  expected  from  a study  such  as 
this.  In  its  analyses  of  the  data  acquired  from  the  Love  Canal  samples,  EPA 
accepted,  in  some  cases,  data  which  had  uncertainties  as  high  as  one  or  two 
orders  of  magnitude  (cf.  Table  1 1 1- 5 , Page  1 1 1—28  of  the  draft  report). 

Criteria  (reviewed  in  Sections  II  and  III  below)  were  established  for  the 
acceptance  or  rejection  of  data  and  were  based  on  laboratory  analyses  of 
quality  assurance  samples.  However,  accuracy  and  precision  goals  for  the 
organic  data  were  not  established  at  the  outset.  The  draft  report  includes 
statements  of  precision  and  accuracy  which  varied  from  one  procedure  to 
another  and  from  one  laboratory  to  another. 


0.  If  inadequacies  in  the  precision  and  accuracy  goals  are  identified  in 
answering  the  above  questions,  will  these  inadequacies  affect  our 
judgment  of  the  quality  of  the  organic  analytical  data? 


The  establishment  of  precision  and  accuracy  goals,  the  training  of 
laboratory  personnel  to  meet  those  goals,  and  the  use  of  quality  control 
charts  based  on  those  predetermined  goals  would  have  improved  data  quality. 


Without  precision  and  accuracy  goals  which  have  been  formulated  on  the  basis 
of  the  overall  goals  of  the  study,  it  is  difficult  to  determine  if  these 
overall  goals  were  actually  met.  Moreover,  the  lack  of  clarity  in  relating 
the  precision  and  accuracy  estimates  of  the  acquired  data  to  the  overall 
conclusions  of  the  report  will  make  decisions  based  on  those  conclusions 
difficult  to  justify.  This  lack  of  clarity  is  illustrated  by  the  following 
excerpt  from  the  discussion  of  air  contamination  on  Page  183  of  the  draft 
report: 

“Even  though  maximum  concentration  levels  are  often  of  considerable 
interest  to  individuals,  because  in  some  way  they  may  be  thought  to 
represent  'worst  case'  estimates  of  environmental  contamination, 
problems  of  interpretation  exist.  The  reason  for  this  is  because  both 
the  occurrence  and  reliability  of  the  obtained  maximum  values  may  be 
plagued  by  measurement  problems.  To  illustrate  this  point,  it  is  often 
the  case  that  maximum  concentrations  are  reported  by  only  one  analytical 
laboratory  and  on  one  particular  date,  whereas  other  analytical 
laboratories  do  not  report  concentration  levels  anywhere  near  such 
maxima  (and  in  some  cases  do  not  even  report  concentration  levels  above 
the  minimum  detection  value)." 

Statements  such  as  these  allude,  in  a general  way,  to  the  lack  of  precision 
and  accuracy  in  the  data  but  are  not  supported  by  quantitative  information 
in  the  draft  report.  They  give  little  guidance  to  the  user  of  the  data  on 
how  the  lack  of  precision  and  accuracy  affects  conclusions  that  might  be 
drawn. 
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II.  Monitoring  Protocols 


Scope:  To  evaluate  the  consistency  of  the  monitoring  protocols  with  the 

goals  and  objectives  of  the  Organic  Analysis  Project  and  with  existing 
state-of-the-art  methodology. 


A.  Have  the  sources  of  the  protocols  used  by  EPA  been  identified?  Are 
these  protocols  well  documented?  Had  they  been  evaluated  and  did  EPA 
consider  these  evaluations  in  their  selection?  Are  there  additional 
references  EPA  should  have  considered  in  the  selection  of  these 
protocol s? 


The  analytical  methodology  used  in  the  study  is  described  in  the  docunent 
"Qua!  ity.  Assurance  Plan:  LC- 1-619-206,  Appendix  B"  by  the  prime  contractor 

for  the  EPA  study,  the  GCA  Corporation. 

Those  methods  which  were  used  in  the  water  analyses.  Methods  608,  624,  and 
625,  had  been,  prior  to  the  initiation  of  the  study,  published  in  the 
Federal  Register.  These  methods  were  in  the  process  of  being  validated  by 
interlaboratory  tests,  and  such  validation  may  now  be  complete.  To  the  best 
of  our  knowledge,  such  validation  was  not  available  at  the  outset  of  the 
Love  Canal  Monitoring  Study  and  hence  could  not  be  used  as  a guide  for 
quality  control.  When  Methods  608,  624,  and  625  were  published  in  the 
Federal  Register  in  December  1979,  EPA  asked  for  and  received  comments  on 

the  methods.  These  comments  have  been  considered  by  EPA,  were  found  in  some 

cases  to  be  germane,  and  if  incorporated  would  have  led  to  improvement  in 
the  clarity  of  presentation  or  in  the  actual  performance  of  the  methods. 
These  modifications  were  not  incorporated  into  the  methods  used  in  the  Love 
Canal  Monitoring  Study.  Nevertheless,  Methods  608,  624,  and  625  have  been 
widely  used  and  in  our  opinion  represent  a reasonable  choice  for  the  study. 

The  methods  used  for  the  air  analyses  are  well  referenced  in  the  draft 
report  and  are  described  in  detail  in  the  aforementioned  Appendix  B of  the 
Love  Canal  Quality  Assurance  Plan.  To  our  knowledge  these  methods  have  not 
been  formally  validated. 

The  sample  preparation  steps  in  the  methods  used  for  soil,  sediments  and 

biota  samples  were  experimental  in  nature  and  had  to  be  developed  as  part  of 

the  Love  Canal  Monitoring  Study.  They  were,  consequently,  unevaluated 
beyond  the  evaluation  appearing  in  the  draft  report.  A nunber  of  the 
protocols  which,  taken  together,  comprised  the  methods  for  the  soil, 
sediments  and  biota  were  still  optional  at  the  time  Appendix  B of  the 
Quality  Assurance  Plan  was  released.  Although  the  methodology  was  fixed  by 
the  end  of  the  study,  it  is  unclear  whether  all  the  data  acquired  for  the 
soil,  sediments  and  biota  were  obtained  using  the  methodology  described  in 
the  draft  report.  9 ]’ 
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There  are  no  fundamentally  different  analytical  techniques  for  the  detection 
and  quantitation  of  organic  compounds  which  would  have  been  more  suitable 
for  the  study  as  planned. 


B.  Does  EPA  discuss  the  precision  and  accuracy  to  be  expected  from  the 
chosen  protocols?  Does  EPA  document  the  minimum  detectable  limits  for 
their  targeted  compounds?  Are  the  precision,  accuracy  and  minimun 
detectable  limits  claimed  by  EPA  reasonable  and  consistent  with  our 
knowledge  of  or  experience  with  the  methodology?  Are  these  claims 
supported  by  reference  to  available  literature? 


Since  none  of  the  methods  used  in  the  study  had  been  subjected  to  a formal 
inter! aboratory  validation  prior  to  the  study,  expectations  concerning  the 
precision,  accuracy  and  minimum  detection  limits  had  to  be  drawn  from  the 
reported  experience  of  analysts  who  had  used  the  various  methods. 

For  air  methods,  EPA  established  limits  of  quantitation  for  targeted 
compounds;  limits  of  detection  were  to  be  determined  by  each  laboratory. 

For  water  methods,  the  limits  of  quantitation  for  targeted  compounds  were  to 
be  determined  by  each  laboratory;  expected  limits  of  detection  were  reported 
in  the  Federal  Register  description  of  the  methods. 

For  many  of  the  protocols,  expectations  of  precision,  accuracy,  and 
detection  limits  were  unavailable  and  were  to  be  determined  during  the 
course  of  the  study  from  the  quality  assurance  data.  The  protocols  did  not, 
in  all  cases,  explain  how  such  determinations  were  to  be  made.  Moreover, 
such  guidance  as  was  given  varied  from  one  environmental  medium  to  another. 
For  example,  precision  was  to  be  determined  for  air  monitoring  methods  from 
data  on  duplicate  measurements  and  sample  splits  and  the  accuracy  from  data 
on  spiked  TENAX  tubes  and  polyurethane  foam  plugs,  while  for  total  organic 
carbon  determinations  in  water,  the  methods  called  for  the  analysis  of 
blanks  and  replicates  but  had  no  requirement  for  the  determination  of 
accuracy. 

Not  all  claims  on  expected  precision,  accuracy,  and  detection  limits  were 
supported  by  appropriate  references.  The  requirement  that  they  be 
determined  by  the  individual  laboratories  during  the  course  of  the  study  is 
correct  and  always  necessary. 


C.  Can  we  identify  targeted  or  non-targeted  compounds  which  the  monitoring 
protocols  would  be  likely  to  miss?  Can  we  recommend  ways  in  which  EPA 
could  justify  the  completeness  of  its  identification  of  compounds  which 
it  has  not  presented  in  the  written  documentation  supplied  to  NBS? 
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In  general,  the  protocols  were  capable  of  detecting  all  targeted  compounds. 
Because  of  the  composition  of  TENAX,  and  the  allowed  lapse  time  between 
collection  and  analysis  of  samples,  the  use  of  TENAX  for  the  analysis  of 
benzene  and  toluene  in  air  is  inappropriate  at  the  levels  encountered.  This 
problem  has  been  recognized  by  EPA  and  is  commented  upon  in  its  draft 
report. 


The  protocols  are  likely  to  miss  certain  non-targeted  compounds.  In 
particular,  compounds  which  are  too  polar  or  not  sufficiently  volatile  will 
not  be  detectable  using  GC-MS. 


D.  Are  the  protocols  as  written  complete?  Do  they  omit  steps  or 

precautions  that  would  be  expected  to  affect  the  identification  or 
quantification  of  certain  organic  compounds?  Were  all  of  the  following 
steps  clearly  described  in  the  protocols,  were  they  complete? 


1. 

Sample  Collection  Techniques 

2. 

Sample  Storage  and  Transportation 

3. 

Chemical  and  Physical  Manipulation  of  the  Samples 

Prior  to  Analysis 

4. 

Calibration 

5. 

Analysi  s 

6. 

Records 

The  protocols  were  sufficiently  complete  in  the  sense  that  they  could  be 
followed  by  competent  analytical  chemists  with  experience  in  organic 
analysis.  The  protocols  for  the  sample  collection,  for  storage  and 
transportation,  and  for  record  keeping  were  satisfactory.  Some  ambiguities 
in  the  remaining  steps  could  have  resulted  in  actual  differences  in  how  the 
protocols  were  executed.  Such  differences  could  introduce  difficulties  when 
comparing  data  from  the  various  laboratories. 

Instances  of  incompleteness  or  ambiguity  in  the  protocols  are  listed  below,  j 

(1)  The  analytical  procedure  for  volatile  organics  on  TENAX  allows  for  the 
use  of  an  internal  standard,  dependent  on  the  availability  of  the 
apparatus  needed  to  perform  spiking  (Quality  Assurance  Plan,  Appendix  B, 
Section  1.2.2C).  This  apparatus  is  not  defined  in  the  procedure  and  the 
addition  of  an  internal  standard  is  optional.  Comparability  of  data 
between  laboratories  requires  comparable  methods  of  internal  standard 
addition. 

(2)  Mass  spectrometri c confirmation  of  gas  chromatography-electron  capture 
(GC-EC)  and  high  performance  liquid  chromatography  (HPLC)  measurements  ; 
for  pesticides  collected  on  polyurethane  foam  plugs  are  to  be  conducted 
on  composited  or  selected  samples.  The  choice  of  these  samples  is  not 
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specified.  Furthermore,  Subsection  F,  Section  1.3.1,  Appendix  B of  the 
Quality  Assurance  Plan  of  this  analytical  procedure  directs  the  analyst 
to  “confirm  results  as  required  by  combined  GC-MS"  but  does  not  specify 
what  constitutes  “as  required." 

(3)  The  analyst  is  given  the  option  of  choosing  from  four  methods,  involving 
different  extraction  schemes  and  packed  or  capillary  columns  for  the 
analysis  of  semi -volatile  organic  compounds  in  water. 

(4)  The  analytical  procedure  for  the  determination  of  total  organic  carbon 
in  water  samples  requires  a chemical  preservation  step  to  be  performed 
in  the  laboratory  within  two  hours  of  collection.  The  QA  procedure 
■submitted  by  JRB  Associates  specifies  that  samples  to  be  analyzed  for 
total  organic  carbon  and  total  organic  halogen  content  are  to  be 
transferred  to  the  GCA  sample  bank  within  four  hours  of  collection. 

(5)  The  analytical  method  for  the  determination  of  pesticides  in  sediments 
calls  for  the  use  of  a procedure  entitled  “Sample  Preparation  and 
Analysis  of  Bottom  Sediments"  with  the  exception  that  part  5 of  the 
■method  has  been  modified  and  part  4 of  the  method  is  to  be  disregarded. 
There  is  no  statement  of  what  is  to  be  done  in  place  of  part  4.  (Part  4 
is  the  section  dealing  with  sample  preparation  and  extraction 
procedures.)  One  could  assune  that  Method  608  is  the  replacement,  but 
this  is  not  specified.  In  this  same  bottom  sediments  procedure,  the  gas 
chromatographic  step  calls  for  proceeding  as  described  in  Section  11. A. 
There  is  a Section  11  of  Method  608  which  deals  with  gas  chromatogr aphy 
but  there  is  no  subsection  11. A. 


E.  Do  the  protocols  adequately  address  the  prevention  of  sample 

contamination  from  impure  reagents,  the  sampling  process,  contaminated 
sample  containers,  etc.? 


With  few  exceptions,  the  protocols  dealt  with  the  problems  of  analytical 
contamination  in  a reasonable  manner.  These  exceptions  included  the 
problems  of  benzene  and  toluene  contamination  of  TENAX  mentioned  in  II. C 
above  and  problems  with  vagueness  or  inconsistency  in  instructions  on  how  to 
correct  for  common  sources  of  analytical  contamination.  For  example,  the 
quality  assurance  procedure  for  the  analysis  of  semi-volatile  compounds  in 
soil  and  sediment  required  that  the  method  blank  not  show  any  signal 
corresponding  to  compounds  on  the  list  of  targeted  compounds,  but  if  common 
laboratory  contaminants  such  as  phthalate  esters  (which  were  on  the  targeted 
list)  are  found,  the  reported  values  for  the  corresponding  compounds  in  the 
environmental  samples  must  be  reduced  by  the  amount  found  in  the  blank. 

This  instruction  is  vague  in  that  levels  of  contamination  for  which  method 
blank  corrections  are  valid  are  not  given. 
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F. 


In  the  documentation  provided  by  EPA,  is  the  intended  use  and  purpose  of 
the  data  from  the  control  samples  described?  Is  this  intended  use 
reasonable?  Could  it  have  been  expected  to  achieve  its  intended 
purpose? 


The  control  samples,  that  is,  samples  from  the  Control  Area,  were  designet 
for  comparison  of  levels  of  contamination  in  the  Declaration  Area  with 
levels  outside  that  area.  (As  such,  they  were  not  intended  as  analytical 
controls  and  should  not  be  interpreted  as  such.  They  are,  within  the 
context  of  a review  of  the  analytical  program,  to  be  treated  as 
environmental  samples,  indistinguishable  from  any  other  environmental 
samples.)  Their  use  and  purpose  are  described  and  the  intended  use  is 
reasonable.  Determination  of  whether  or  not  they  were  sufficient  in  number 
and  kind  to  have  achieved  their  intended  purpose  is  beyond  the  charge  from 
EPA  to  NBS. 


G. 


Are  there  any  means,  not  addressed  above  but  available  to  EPA  at  the 
beginning  of  the  study,  by  which  EPA  could  have  improved  upon  the 
selection,  description  or  justification  for  use  of  the  selected 
monitoring  protocols? 


Improvements  in  the  methodology  have  been  addressed  in  the  sections  above, 
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[II.  Quality  Assurance  Protocols 

Scope:  To  evaluate  the  consistency  of  the  quality  assurance  protocols  with 

the  accuracy  and  precision  objectives  of  the  study  and  to  evaluate  the 
adequacy  of  these  protocols  to  maintain  quality  control  over  the  reported 
data. 


A.  Was  the  quality  assurance  plan  sufficient  to  maintain  quality  control? 

If  not,  what  were  the  major  flaws  in  the  plan?  Were  there  any  necessary 
elements  missing? 


A major  feature  of  the  Love  Canal  Monitoring  Study  was  the  quality  assurance 
program  for  the  study.  The  prime  contractor,  GCA,  prepared  an  extensive 
quality  assurance  plan,  and  all  but  one  of  the  participating  subcontractors 
prepared  quality  assurance  plans  in  response  to  the  GCA  plan.  These  plans 
contained  a large  variety  of  quality  assurance  measures  including  document 
and  report  control,  laboratory  control  standards,  replicate  analyses, 
internal  standards,  calibration  checks,  and  surrogate  standard  additions. 

Quality  assurance  must  be  recognized  as  consisting  of  two  factors:  quality 

control  and  quality  assessment.  The  quality  control  aspects  of  the  plans 
were  generally  adequate,  especially  those  portions  which  dealt  with  the 
collection  and  handling  of  samples  and  maintenance  of  records,  including 
chain  of  custody.  The  Quality  Assurance  Plan  was  deficient  in  the  manner  in 
which  it  addressed  quality  assessment.  The  interrelation  of  individual 
items  and  the  manner  in  which  quality  assurance  data  was  to  be  analyzed  and 
used  were  not  clearly  described. 

Protocols  varied  between  media.  Examples  of  this  variability  from  Section  7 
of  the  Quality  Assurance  Plan  will  be  described  in  the  following  paragraphs. 

(1)  For  air  samples  collected  on  TENAX  cartridges,  the  use  of  an  internal 
standard  was  optional  and  this  standard  when  used  was  to  be  added  to  the 
TENAX  cartridge?  TnFernal  standards  should  have  been  mandatory.  For 
air  samples  collected  on  polyurethane  foam  plugs,  an  internal  standard 
was  mandatory,  but  was  added  to  the  sample  extract.  Moreover,  the 
procedure  stated  "if  a suitable  compound  can  be  determined  a single 
internal  standard  will  be  added  to  each  sample  extract."  The  standard 
should  have  been  specified  and  should  have  been  added  as  early  as 
possible  in  the  analytical  procedure.  For  the  determination  of  dioxin 
(TCDO)  in  air  samples,  chlorine-37  labeled  TCDO  was  added  to  each  sample 
as  an  internal  standard.  This  was  appropriate. 

(2)  For  those  air  samples  collected  on  polyurethane  foam  plugs,  a three 
point  calibration  curve  of  all  compounds  was  required  each  day;  a 
single-concentration,  mixed  standard  was  required  after  every  fifth 
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sample;  and  one  spiked  foam  plug  was  to  be  analyzed  per  analysis  batch 
per  day.  For  those  air  samples  collected  on  TENAX,  only  a single 
calibration  check  was  required  per  eight  hours  of  analysis.  No 
discussion  is  presented  describing  why  the  two  protocols  were  so 
different. 

(3)  For  the  determination  of  semi -vol ati 1 e organic  compounds  in  water, 
deuterated  internal  standards  were  required  (but  the  draft  report 
contains  no  mention  of  their  use).  For  the  determination  of  volatile 
organic  compounds  in  water,  internal  standards  (non-deuterated)  were 
optional.  For  the  determination  of  pesticides  in  water,  the  analyst  is 
referred  to  Section  7 of  Method  608  which  recommends  the  use  of 
fortified  samples.  The  use  of  internal  standards  should  have  been 
required. 

(4)  For  the  determination  of  volatile  and  semi -vol  ati  1 e organic  compounds  in 
water,  surrogate  compound  additions  were  required.  Recoveries  of 
surrogate  compounds  were  to  fall  within  specified  ranges.  (The  ranges 
described  in  the  draft  report  were  substantially  different  from  the 
ranges  in  the  Quality  Assurance  Plan.)  For  pesticides,  the  use  of 
fortified  samples  was  recommended  (by  reference  to  Method  608). 

(5)  For  the  determination  of  pesticides,  volatile  organic  compounds,  and 
semi -vol ati le  organic  compounds  in  water,  laboratory  control  standards 
were  required.  Control  limits  were  established  for  the  recoveries  of 
the  laboratory  control  standards.  For  pesticides,  the  control  limits 
were  specified  as  "±2  standard  deviations."  For  volatile  and  semi- 
volatile organic  compounds,  the  control  limits  were  specified  as  ranges  j 
of  "percent  of  true  value"  for  individual  compounds. 

(6)  For  the  analysis  of  pesticides  in  soils  and  house  dust,  a recovery  check 
on  a spiked  sample  using  a list  of  specified  pesticides  was  required. 

The  same  recovery  check  was  not  part  of  the  sediment  procedure  and 
should  have  been. 

j 

(7)  For  the  analysis  of  total  organic  carbon  and  total  organic  halogen 
content  in  water,  specifications  were  presented  for  method  blanks  and 
replicate  analyses,  but  none  for  a laboratory  control  standard,  an 
internal  standard  or  a surrogate  standard;  hence,  there  was  no  accuracy 
check. 

In  addition,  control  limits  associated  with  many  of  the  quality  assurance 
procedures  were  to  be  determined  during  the  course  of  the  study.  These 
limits,  if  they  were  determined,  have  not  been  reported.  Performance  as 
determined  from  calibration  check  samples  and  performance  evaluation  samples 
was  to  be  evaluated  as  the  study  progressed.  This  evaluation  process  was 
not  described  in  the  Quality  Assurance  Plan. 

i 

The  above  remarks  indicate  the  quality  assurance  plans  were  quite  extensive 
and  detailed,  but  were  lacking  in  consistency  from  one  medium  to  the  next 
and  lacking  in  detail  on  how  the  quality  assessment  was  to  be  performed. 
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B.  Were  criteria  established  for  the  rejection  or  acceptance  of  data?  Were 
these  criteria  reasonable,  and  if  not,  how  could  they  have  been 
improved? 


Two  levels  of  review  for  rejection  or  acceptance  of  data  were  included  in 
the  Love  Canal  Monitoring  Study.  The  Quality  Assurance  Plan  called  for 
contract  laboratories  to  monitor  their  own  performance  and  to  invalidate 
data  according  to  certain  criteria.  Laboratory  performance  was  to  be 
reviewed  by  the  prime  contractor  and  by  EPA.  Criteria  for  acceptance  or 
rejection  by  the  prime  contractor  or  by  EPA  of  data  already  validated  by  the 
contract  laboratories  were  not  part  of  the  Quality  Assurance  Plan,  but  were 
described  in  the  draft  report.  Furthermore,  these  criteria  were  not 
consistent  with  those  in  the  Quality  Assurance  Plan,  as  will  be  described  in 
the  following  paragraph.  In  contrast  to  the  statement  on  page  64  of  the 
draft  report,  the  Quality  Assurance  Plan  did  not  require  the  contract 
laboratories  to  maintain  and  submit  quality  control  charts. 

The  criteria  used  by  EPA  for  acceptance  or  rejection  of  data  differed  from 
the  criteria  specified  for  the  contract  laboratories  in  the  Quality 
Assurance  Plan.  For  example.  Table  7.2.1  of  the  Quality  Assurance  Plan 
specifies  control  limits  for  the  recovery  of  surrogate  compounds  in  water  of 
81  to  100  percent  for  f 1 uorobenzene,  88  to  118  percent  for  p-bromo- 
f luorobenzene,  18  to  58  percent  for  2-f luorophenol , 33  to  8l?  percent  for  1- 
f luoronaphthalene,  and  34  to  98  percent  for  4 ,4' -dibromooctaf 1 uor o- 
biphenyl.  Table  III-5,  Page  III- 28  of  the  draft  report  indicates  there  were 
no  upper  control  limits  for  the  recovery  of  these  five  surrogates  and  that 
EPA  invalidated  all  analyses  for  which  more  than  one  of  the  surrogates  was 
lower  than  the  lower  control  limits  of  68  percent,  60  percent,  1 percent, 

2.8  percent,  and  8.3  percent  respectively  for  the  five  compounds  mentioned 
above. 

The  EPA  criteria  for  validation  of  data  were  less  stringent,  in  practice, 
than  the  criteria  in  the  Quality  Assurance  Plan.  Whether  this  indicates  the 
plans  were  unrealistic  or  the  implementation  of  the  Quality  Assurance  Plan 
was  inadequate  cannot  be  determined. 


C.  Were  the  quality  assurance  plans  of  the  individual  contractors  uniform 
and  consistent  with  each  other  and  with  the  overall  Quality  Assurance 
Plan? 


The  quality  assurance  plans  of  the  individual  contractors  followed  an 
essentially  consistent  outline  which  was  appropriate  to  the  study.  The 
individual  plans,  however,  differed  in  detail. 

Not  all  quality  assurance  plans  were  available  or  complete  when  NBS  began 
its  review.  In  particular,  to  our  knowledge,  no  plan  was  filed  by  Research 
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Triangle  Institute  (RTI)  which  was  one  of  the  three  QA  contractors  and  which 
had  responsibility  for  the  spiking  of  the  TENAX  cartridges  used  in  the  QA 
program  for  the  analyses  of  air  samples.  The  QA  plans  from  TRW,  one  of  the 
laboratories  performing  analyses  for  organic  compounds  in  water,  and  Wright 
State  University,  the  only  laboratory  performing  dioxin  analyses,  were 
received  late  by  NBS  and  it  is  not  clear  whether  these  plans  had  received 
approval  from  EPA  prior  to  the  initiation  of  the  Love  Canal  Monitoring 
Study.  The  plans  received  from  CompuChem/Mead  and  Ac curex  Corp.  contained 
notations  that  additional  information  had  been  requested.  This  information 
was  received  by  NBS  well  after  completion  of  the  chemical  analyses.  It  is 
not  clear  whether  these  additions  were  included  in  their  QA  plans  or  were 
descriptions  of  practices  which  had  been  followed.  The  plans  from  PJB  and 
Advanced  Environmental  Systems  were  lacking  in  a nunber  of  elements  required 
by  the  overall  QA  plan  of  the  prime  contractor,  but  no  indication  was  given 
whether  revised  plans  or  additional  information  had  been  requested.  The  QA 
plan  for  Southwest  Research  Institute  provided  no  information  on  the 
analytical  procedure  or  the  quality  control  procedure  for  the  preparation  of 
the  polyurethane  foam  plugs  which  were  used  in  the  air  analyses.  The  plan 
from  Battel le  Columbus  Laboratories  lacked  a summary  of  analytical  methods, 
equipment  maintenance,  and  trouble-shooting  procedures.  Finally,  none  of 
the  QA  plans  addressed  the  identification  of  the  20  most  abundant,  non- 
targeted  compounds. 

The  quality  assurance  plans  should  always  be  signed  and  dated  by  the 
submitting  1 aboratory  and  the  approving  authority  before  work  begins.  This 
was  not  done  consistently  in  the  Love  Canal  Monitoring  Study. 


D.  Was  the  number,  nature  and  frequency  of  quality  control  samples  or 
performance  audit  samples  appropriate  to  achieve  monitoring  goals,  to 
take  corrective  action,  and/or  to  maintain  quality  control? 


The  number,  nature,  and  frequency  of  quality  control  samples  as  required  by 
the  Quality  Assurance  Plan  should  have  been  adequate  to  maintain  quality 
control.  Evidence  exists,  however,  that  quality  control  was  not  maintained 
in  some  cases.  The  Quality  Assurance  Plan  did  not  anticipate  the  large 
number  of  samples  which  would  be  found  to  contain  no  levels  of  contamination 
above  the  limits  of  detection.  When  a large  number  of  samples  are  expected 
to  contain  concentrations  below  the  detection  limit,  sample  splits  and 
duplicate  samples  are  of  limited  usefulness.  These  difficulties  will  be 
discussed  further  in  Section  V dealing  with  implementation. 
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E.  Did  the  plan  adequately  and  clearly  prescribe  corrective  action? 


Instructions  from  EPA  and  the  prime  contractor  to  the  contract  laboratories 
for  corrective  action  (CA)  were  clearly  and  adequately  described  in  the 
overall  Quality  Assurance  Plan  with  two  exceptions:  (1)  The  overall  plan 

did  not  require  contract  laboratories  to  maintain  and  submit  quality  control 
charts.  (2)  There  existed  ambiguity  in  the  description  of  the  mechanism  for 
reporting  corrective  action  to  the  prime  contractor.  Figure  11-1,  Page  11-3 
of  the  Quality  Assurance  Plan,  is  an  example  of  a form  which  was  to  be 
completed  and  submitted  describing  corrective  action  taken  whenever  loss  of 
quality  control  was  identified.  On  Page  11-4,  however,  it  was  stated 
"subcontractors  are  not  required  to  follow  this  system  exactly  or  to  use  the 
pictured  CA  form."  For  such  a large  and  complex  study  conducted  in  such  a 
short  period  of  time,  a uniform  system  should  have  been  required.  On  the 
basis  of  the  written,  corrective-action  procedures  appearing  in  the 
individual  subcontractor  QA  plans,  it  is  apparent  that  corrective-action 
procedures  were  not  consistently  adequate.  For  example,  1 1 T stated 
corrective  action  (unspecified)  would  be  implemented  as  required  but  did  not 
state  that  it  would  report  the  need  for  corrective  action  to  GCA.  PEDCO 
also  gave  only  a vague  statement  of  the  nature  of  corrective-action 
procedures  and  the  documentation  of  corrective  action. 
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IV.  Performance  of  the  Monitoring  Program 

Scope:  To  evaluate,  from  the  information  provided  by  EPA,  whether  or  not 

the  implementation  of  the  monitoring  program  met  the  goals  and  objectives  of 
the  Organic  Analysis  Project  and  were  consistent  with  proper  implementation 
of  the  monitoring  protocols  and  with  commonly  accepted  good  laboratory 
practices. 


A.  What  evidence  does  EPA  present  to  support  claims  of  proper 

implementation  of  the  monitoring  protocols?  Is  the  evidence  presented 
that  the  contract  laboratories  were  capable  of  performing  their  assigned 
tasks  convincing? 


Based  on  the  material  reviewed,  the  sample  collection,  storage  and 
transportation,  document  control/chain  of  custody  procedures,  and  data 
management  were  generally  implemented  according  to  plan.  Field  and  quality 
assurance  samples  were  distributed  according  to  plan.  An  acceptable 
fraction  of  the  samples  was  analyzed.  Some  difficulties  with  the  analysis 
of  the  samples  were  encountered  and  these  difficulties  are  discussed  in  the 
remainder.of  this  Section  and  in  Section  V below. 


B.  Is  there  any  reason  to  believe  that  significant  concentrations  of 
targeted  or  non-targeted  compounds  were  missed  in  the  analyses  of  the 
samples?  ' What  additional  information  should  EPA  provide  to  justify 
claims  that  compounds  were  correctly  identified  and  that  significant 
concentrations  of  targeted  or  non-targeted  compounds  were  not  missed? 


The  GC-MS  audit  of  computer  records  indicated  that  most  non-targeted 
compounds  were  probably  missed.  Pages  VI-12  to  13  of  the  draft  report 
state: 

"In  most  of  the  22  samples  [of  the  80  samples  audited]  containing  non- 
target compounds,  the  audit  laboratory  reported  finding  1 or  2 compounds 
while  the  analytical  laboratory  reported  none.  There  were  5 samples 
audited  where  6 to  20  compounds  were  reported  by  the  audit  laboratory 
but  none  were  reported  by  the  analytical  1 aboratory. . . ." 

The  audit  also  indicated  inconsistency  between  laboratories  in  the 
determination  of  the  presence  of  targeted  compounds  when  the  concentration 
of  those  compounds  was  near  the  detection  limit. 

Pages  II 1-14  to  II 1-16  contain  a discussion  of  the  limits  of  detection  and 
quantitation  for  organic  compounds  in  water.  Table  III-l,  Page  1 11-15, 
contains  a summary  of  the  limits  of  quantitation,  and  the  accompanying 
discussion  states: 


"There  was  considerable  variance  among  the  analytical 
laboratories  in  method  quantitation  limits  for  a given  analyte, 
and  the  data  suggest  that  some  laboratories  were  not  operating 
consistently  at  the  state-of-the-art  possible  with  the  methods. 

This  is  neither  unusual  nor  unexpected." 

Comparison  of  the  numbers  in  Table  III-l  with: 

(a)  Table  II-l,  Page  1 1-5  which  contains  the  range  of  median  values  of 
compounds  found  in  drinking  and  surface  waters  in  various  industrial 
locations, 

(b)  Table  1 1-4,  Page  1 1-8  which  contains  values  of  volatile  organic 
compounds  found  in  drinking  water  in  a number  of  cities,  and 

(c)  Table  1 1-9,  Page  11-34  which  contains  values  of  selected  volatile 
organic  compounds  found  in  national  surveys  of  water  supplies, 

indicates  that  the  limits  of  quantitation  for  volatile  organics,  determined 
by  three  of  the  five  laboratories  performing  water  analyses  in  the  Love 
Canal  Monitoring  Study,  were  an  order  of  magnitude  higher  than  values 
reported  by  laboratories  participating  in  the  studies  quoted  in  Appendix  II 
of  the  EPA  draft  report.  Also,  for  the  volatile  organics,  the  limits  of 
quantitation  for  the  five  contract  laboratories  were  1,  2,  4,  8 and  9 times 
higher  than  those  of  EPA's  own  laboratory  in  Cincinnati.  For  the 
semivolatile  organics,  little  data  exist  against  which  to  compare  the 
results  of  this  study,  but  the  limits  of  quantitation  for  the  four  contract 
laboratories  were  1,  8,  9,  and  14  times  higher  than  those  of  the  EPA 
laboratory.  In  its  draft  report,  EPA  recognizes  the  variability  in  limits 
of  quantitation  reported  by  the  contract  laboratories.  This  variability 
will  make  the  Love  Canal  monitoring  data  difficult  to  interpret. 

Tlie  limits  of  quantitation  set  for  the  air  monitoring  laboratories  appear  to 
be  adequate  for  quantifying  levels  of  compounds  which  have  been  reported  for 
some  U.S.  cities. 


C.  Is  there  any  reason  to  believe  that  significant  contamination  of  the 
samples  from  impure  reagents,  the  sampling  process,  or  contaminated 
tools,  or  containers  may  have  occurred?  Does  EPA  present  evidence  that 
this  has  not  occurred?  Is  it  convincing  and  if  not,  what  additional 
documentation  is  needed? 


EPA  has  presented  evidence  that  artifactual  contamination,  that  is, 
contamination  of  the  samples  from  impure  reagents,  the  sampling  process,  or 
from  contaminated  tools  or  containers,  was  a problem  for  certain  samples  and 
for  identified  analytes. 

(1)  For  those  air  samples  collected  on  TENAX,  significant  blank  problems 
associated  with  benzene,  toluene,  and  1,1,2,2-tetrachloroethyl ene  were 
encountered.  In  Table  V-4,  Page  V-13  of  the  draft  report,  EPA  indicates 
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the  frequency  and  concentrations  of  these  compounds  found  on  blank  TENAX 
cartridges.  The  accompanying  text  states: 

"To  be  relatively  certain  that  an  obtained  single  value 
was  not  due  to  blank  contamination,  the  field  concentration 
should  be  greater  than  three  standard  deviations  above  the  mean 
values  reported  in  Table  V- 4 for  these  three  compounds." 


From  the  data  in  Table  V- 4,  "three  standard  deviations  above  the  mean 
values"  were  7,  29,  and  22  micrograms/cubic  meter  for  benzene,  toluene 
and  1,1 ,2 ,2-tetrachloroethylene  respectively.  According  to  the  draft 
report  analytical  values  obtained  for  these  compounds  below  these  levels 
would  be  suspect.  (Averages  of  several  analyses  would  have  somewhat 
lower  limits  depending  on  the  number  of  samples  included  in  the 
analyses.)  In  Section  4.2.6,  Pages  167  to  195  of  the  draft  report,  EPA 
describes  the  results  of  the  air  monitoring  program.  The  only 
significant  differences  between  the  Declaration,  Canal  and  Control  Areas 
are  the  concentrations  of  o-chlorotoluene,  o-dichlorobenzene,  and 
chlorobenzene.  Yet,  for  tFe  remainder  of  tHe  discussion,  only  benzene, 
toluene  and  1,1,2,2-tetrachloroethylene  are  discussed.  We  question  the 
appropri ateness  of  using  these  three  compounds,  for  which  blank  problems 
call  into  question  the  reliability  of  their  determination,  as  "typical" 
examples.  The  median  concentration  plots  in  Figures  54,  55  and  56  on 
Pages  188  to  190  were  chosen  to  illustrate  the  lack  of  pattern  in  the 
distribution  of  these  compounds,  but  the  maximum  values  for  these 
medians  are  below  the  above-mentioned  limits  on  TENAX  blanks. 

(2)  On  Page  1 1 1-29.  the  contamination  of  bedrock  B wells  by  use  of  ordinary  j 
hydrant  water  during  the  drilling  process  is  described.  This  led  to  the 
exclusion  of  "28  ground-water , Method  624  samples"  from  the  validated 
data  base.  Chloroform  was  identified  as  the  contaminant  and  its  source 
was  attributed  by  EPA  to  improper  purging  of  the  wells,  but  no 
quantitative  data  were  presented  to  support  this  claim.  A total  of  340 
bedrock  well  samples  were  included  in  the  validated  sample  set.  The 
effect  of  excluding  the  28  samples  on  the  conclusions  drawn  by  EPA  is 
not  discussed  in  the  draft  report. 


(3)  On  Pages  1 1 1-29  to  31,  the  contamination  of  water  samples  by  methylene 
chloride,  b i s( 2-ethyl  hexyl ) phthalate,  and  dibutyl  phthalate  is 
described.  This  contamination  resulted  from  the  presence  of  these 
chemicals  in  the  atmosphere  of  the  analytical  laboratories  and  from 
plastic  containment  materials.  As  a result,  all  reports  of  these  three 
analytes  were  removed  from  the  validated  data  set.  The  same  problem  was 
encountered  with  the  sediment  and  soil  analyses. 


With  the  exception  of  item  (1)  above,  the  incidence  of  artifactual 
contamination  should  present  no  problem  so  long  as  conclusions  drawn 
concerning  Love  Canal  are  based  on  patterns  observed  for  groups  of  chemicals 
and  not  for  individual  substances.  The  above  examples  of  contamination  were 
discovered  because  so  many  of  the  samples  which  were  collected  had  no 
concentrations  of  analytes  above  the  detection  limit.  For  the  same  reason, 
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it  is  unlikely  there  are  other  cases  of  widespread,  but  as  yet  undetected, 
artif actual  contamination  by  targeted  compounds. 


0.  Is  there  any  evidence  to  indicate  that  participating  laboratories  were 
incapable  of  adequately  following  the  protocols  or  did  not  adequately 
follow  the  protocols?  Is  there  any  evidence  of  carelessness  in 
execution  of  any  of  the  steps  from  sample  collection  to  analysis? 


As  part  of  the  quality  assurance  program,  site  visits  were  made  to  the 
participating  laboratories  and  reports  of  these  visits  were  reviewed  by  NBS. 
The  reports  indicated  that  most  laboratories  were  having  some  difficulties 
at  the  outset,  and  a few  were  having  great  difficulties.  In  particular, 
some  of  the  laboratories  were  experiencing  difficulties  in  adjusting  to  the 
use  of  capillary  columns.  No  follow-up  visits  were  made,  or  if  they  were, 
reports  of  such  follow-up  visits  were  not  provided.  It  is  not  known  if 
appropriate  corrective  action  was  taken  in  all  cases. 

The  data  in  Table  6 on  Page  110  of  the  draft  report  are  inconsistent  with 
the  number  of  targeted  compounds  which  were  actually  sought  in  water, 
sediment  and  soil  samples.  Table  6 contains  a summary  of  the  frequency  of 
detection  of  substances  in  the  validated  Love  Canal  samples.  The  table  also 
contains  the  "number  of  substances  (analytes)  measured"  and  the  number  of 
samples  which  were  analyzed.  Dividing  the  number  of  analytes  by  the  number 
of  samples  gives  the  average  number  of  analytes  per  sample.  These  average 
numbers  are,  for  the  air  analyses,  consistent  with  the  number  of  targeted 
analytes  listed  in  Table  1-2  on  Page  1-4.  For  the  water  and  sediment 
analyses,  these  average  numbers  range  from  20  to  30  for  almost  all  of  the 
samples  analyzed,  whereas  Table  1-1  on  Pages  1-1  and  1-2  indicates  that  135 
analytes  were  to  be  sought.  Similarly,  the  average  number  of  analytes  per 
sample  of  drinking  water  obtained  from  Table  12,  Page  196,  is  25  to  28. 

This  inconsistency  should  be  corrected. 

Performance  qualification  of  all  the  laboratories  prior  to  initiation  of  the 
study,  as  was  done  for  air  monitoring  subcontractors,  would  have  diminished 
the  number  of  problems  encountered  but  could  have  delayed  the  initiation  of 
the  study.  This  has  been  acknowledged  by  EPA. 


E.  Does  EPA,  in  its  report,  adequately  document  the  performance  of  the 

contract  laboratories?  If  not,  what  additional  documentation  should  EPA 
provide? 


Appendix  III  of  the  draft  report  does  not  adequately  document  the 
performance  of  the  water  monitoring  laboratories.  Four  types  of  performance 
checks  were  used.  Laboratories  analyzed  laboratory  control  standards  (LCS) 
and  quality  control  (QC)  samples  daily,  performance  evaluation  (PE)  samples 
monthly  and  surrogates  for  every  analysis.  The  laboratory  control  standards 
will  be  discussed  in  Section  V.B.  The  QC  and  PE  samples  were  concentrates 
of  selected  compounds  in  organic  solvents  which  were  added  to  reagent  water 
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and  analyzed.  The  identities  of  the  compounds  in  the  QC  samples  may  or  may 
not  have  been  known  to  the  performing  laboratories;  the  concentrations  were 
unknown.  The  identities  and  concentrations  of  compounds  in  the  PE  samples 
were  unknown  to  the  performing  1 aboratories.  The  surrogates  were  known 
amounts  of  known  compounds  added  to  each  sample.  No  sunmary  of  laboratory 
performance  on  QC  samples  is  given  in  the  draft  report  and  should  be.  The 
performance  on  QC  and  PE  samples  was  not  used  to  determine  precision  and 
accuracy  for  the  methods  but  should  have  been  used,  at  least,  to  confirm  the 
estimates  of  precision  and  accuracy. 

Performance  on  PE  samples  is  summarized  in  Table  II 1-4  on  Page  II 1-22  of  the 
draft  report  as  percentages  of  analyses  submitted  by  the  contract 
laboratories  which  were  judged  "acceptable."  The  definition  of  "acceptable" 
is  not  given  and  should  be.  The  percentage  of  all  analyses  of  PE  samples 
found  to  be  unacceptable  cannot  be  determined  because  the  numbers  of 
analyses  corresponding  to  the  entries  in  Table  1 1 1- 4 are  not  given.  Thus, 
it  is  not  possible  to  compare  the  performance  on  PE  samples  with  the  overall 
performance,  i.e.,  the  percentage  of  Love  Canal  data  which  was  validated. 

The  performance  on  PE  samples  should  be  consistent  with  the  data  validation 
process  and  this  consistency  should  be  documented  and  discussed.  As  Table 
II 1-4  now  stands,  it  appears  that  the  number  of  PE  sample  analyses  judged  as 
unacceptable  (ranging  from  0 to  75%)  is  considerably  larger  than  the  number 
of  samples  invalidated  on  the  basis  of  surrogate  recoveries  (17  samples). 

This  apparent  inconsistency  should  be  clarified. 

i 

The  documentation  of  the  performance  of  the  air  monitoring  laboratories 
appearing  in  Appendix  V of  the  draft  report  is  generally  acceptable  though 
Table  V-5  on  Page  V-20  would  be  more  readily  interpretable  if  it  contained 
the  ranges  of  the  concentrations  of  the  compounds  to  which  the  standard 
deviations  pertained.  Inconsistencies  exist  in  the  main  body  of  the  draft  j 
report  in  the  description  of  the  performance  of  the  air  monitoring 
laboratories.  On  Page  174,  is  the  statement: 

"The  analytical  accuracy. . .for  the  TENAX  samples  was  within  ±10  percent 
(as  indicated  by  analytical  recovery  of  substances  purposely  introduced 
into  samples  at  concentration  levels  known  only  to  EPA) ." 

On  Page  184  is  the  statement: 

"As  can  be  seen  from  the  results  displayed  in  Figures  52  and  53, 
variability  in  concentration  levels  were  observed  over  time.  As  noted 
above,  however,  much  of  the  variability  observed  in  the  sample  results 
could  be  accounted  for  by  the  systematic  differences  in  analytical 
laboratories  and  by  measurement  error." 

The  variability  in  Figures  52  and  53  ranges  from  0 to  40  yg/m3,  well  beyond 
the  ±10%  "accuracy"  claimed  on  Page  174.  Moreover,  there  is  no  quantitative 
justification  given  of  the  statement  "...could  be  accounted  for  by  the 
systematic  differences  in  analytical  laboratories  and  by  measurement  error." 
These  statements,  as  they  stand,  appear  to  be  in  conflict  in  their 
description  of  the  performance  of  the  air  monitoring  laboratories.  i 
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The  documentation  of  laboratory  performance  would  be  improved  if  the 
performance  of  individual  laboratories  on  all  of  the  different  classes  of 
quality  assurance  samples  were  sunmari zed,  tabulated,  and  compared  with 
criteria  established  for  performance  in  the  Quality  Assurance  Plan. 
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V.  Performance  of  the  Quality  Assurance  Program 


Scope:  To  evaluate  from  the  information  provided  by  EPA  whether  or  not  the 

implementation  of  the  quality  assurance  program  was  adequate  to  maintain 
quality  control  of  the  Organic  Analysis  Project  and  whether  or  not  accuracy 
and  precision  claims,  if  given,  have  been  convincingly  justified. 


A.  Were  the  quality  assurance  samples  sufficient  in  kind,  nunber  and 
reliability  to  test  adequately  the  performance  of  the  participating 
laboratories?  What  sources  of  measurement  error,  if  any,  might  EPA  have 
missed  or  have  had  the  most  difficulty  in  evaluating? 


In  general,  we  believe  the  quality  assurance  samples  were  sufficient  in 
kind,  number  and  reliability  to  test  adequately  the  performance  of  the 
participating  1 aboratori es . They  should  also  have  been  sufficient  to 
maintain  quality  control  provided  appropriate  corrective  action  was  taken. 
The  large  nunber  of  samples  containing  no  analytes  above  the  limits  of 
quantitation  limited  the  usefulness  of  replicates  for  evaluating  the 
comparability  of  analyses  performed  in  different  laboratories  (e.g., 
estimating  i nterl aboratory  precision). 

* 

The  measurement  errors  most  difficult  to  evaluate  are  those  associated  with 
the  soil,  sediment  and  biota  samples.  The  quality  control  samples  used  for 
the  monitoring  of  these  samples  were  water  samples  which  do  not  duplicate 
the  matrix  effects  presented  by  soils,  sediments  and  biota.  Thus,  not  all 
of  the  steps  in  the  analytical  methodology  were  tested  by  the  quality 
assurance  program.  This  is  acknowledged  by  EPA  in  its  draft  report. 


B.  Has  the  rejection  or  acceptance  of  data  from  a participating  laboratory 
been  clearly  explained  and  justified?  Can  we  suggest  ways  in  which  the 
rejection  of  data  might  distort  the  representativeness  of  the  data? 


The  rejection  of  data  has  not  been  clearly  described  and  adequately 
justified. 

m i 

The  fractions  of  samples  rejected  for  various  reasons  in  arriving  at  the 
final  set  of  validated  samples  are  sunmarized  in  Table  1 on  Page  11  of  the 
draft  report. 

(1)  In  Table  1 EPA  indicates  410  samples  out  of  2457  water  samples  which 
were  analyzed  were  rejected.  The  extensive  discussion  on  the  data 
validation  process  for  water  samples  on  Pages  II 1-25  to  29  states  17  of i* 
the  Method  624  and  625  samples  were  rejected.  Discussions  for  the 
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other  analytes  are  vague  but  indicate  similarly  small  numbers  of 
rejected  samples.  Therefore,  it  appears  about  300  samples  were 
rejected  without  a corresponding,  satisfactory  explanation  in  the  draft 
report. 

(2)  The  percentage  figures  in  the  fourth  and  fifth  columns  of  Table  1 do 
not  agree  with  the  totals  listed  in  the  left  three  columns.  Moreover, 
they  do  not  agree  with  the  numbers  of  rejected  samples  appearing  in  the 
appendices.  For  example,  from  Table  1,  a comparison  of  samples 
analyzed  with  samples  validated  for  soil,  sediments  and  biota  give  27, 
4,  and  8 samples  rejected,  presumably  on  the  basis  of  the  validation 
procedure.  Appendix  IV,  Page  IV-23  reports  these  numbers  as  9,  7,  and 
7 respectively. 

(3)  The  footnote  to  Table  1 should  apply  to  the  column  headed  "Percent 
Other"  and  not  "%  Rejected  by  QA/QC." 


Beyond  these  observations,  the  following  difficulties  with  the 

justifications  of  the  rejection  or  acceptance  of  data  were  encountered: 

(1)  Page  II 1-24  contains  a discussion  of  the  errors  which  could  result 
from  holding  samples  to  be  analyzed  by  Method  624  beyond  the 
maximum  allowable  holding  time  of  14  days.  Experiments  to  determine 
the  effects  of  holding  time  were  performed  using  samples  containing  100 
micrograms/liter  concentrations  of  selected  analytes.  The  report 
should  address  and  justify  the  significance  of  these  experiments  to 
samples  containing  10  micrograms/liter,  that  is,  concentrations  near 
the  limits  of  quantitation. 

(2)  The  discussion  on  data  validation  procedures  (Pages  II 1-25  to  33)  and 
on  estimates  of  data  accuracy  (Pages  1 11-36  to  38)  for  the  water 
monitoring  effort  should  contain  a description  of  the  percent 
recoveries  for  the  analytes  studied.  The  reporting  of  data  without 
correction  for  the  lack  of  complete  recovery  of  particular  analytes  has 
been  justified  on  the  basis  that  the  analytical  methods  do  not  provide 
a procedure  to  correct  for  these  losses.  Recovery  information  is 
critical  for  proper  assessment  of  data  resulting  from  the  Love  Canal 
Study.  If  the  mean  recovery  for  a particular  analyte  were  50%,  then, 
on  the  average,  all  data  for  that  particular  analyte  would  tend  to  be 
low  by  a factor  of  two.  This  could  be  important  when  comparing  the 
distributions  of  two  different  analytes.  The  only  indication  of  such 
recoveries  is  given  in  Table  II 1-5  on  Page  II 1-28.  These  data  were 
derived  by  EMSC  and  are  probably  not  representative  of  the  performance 
of  the  contract  laboratories. 

(3)  The  description  of  the  rejection  of  the  28  ground-water.  Method  624 
samples  from  bedrock  B wells  should  be  clarified.  How  did  the 
Environmental  Research  Laboratory  in  Ada,  Oklahoma  determine  these 
wells  were  not  purged  adequately  prior  to  sampling? 
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(4)  As  discussed  in  Section  III.B  above,  criteria  were  established  by  EPA 
for  validation  of  data  by  the  water  monitoring  laboratories  based  on 
recoveries  of  surrogate  compounds.  EPA  used  the  recoveries  of  the 
same  compounds  but  substantially  wider  acceptance  limits  for  its 
validation  of  sample  data  which  should  have  been  rejected  (as  required 
by  the  Quality  Assurance  Plan)  by  the  contract  laboratories.  These 
wider  limits  were  based  on  a statistical  analysis  of  the  recoveries 
measured  by  EMSC  in  5%  of  the  water,  soil  and  sediments  samples.  The 
reasons  why  wider  acceptance  limits  were  used  are  not  discussed  in  the 
draft  report.  A comparison  between  the  number  of  samples,  for  which 

the  data  would  have  been  validated  had  the  original  control  limits  been  , 
used,  and  the  final  number  of  validated  samples  would  likely  give 
greater  insight  into  the  quality  of  the  water  data  than  the  tabular 
data  presented  in  the  draft  report.  As  mentioned  in  Section  III.E 
above,  laboratory  performance  on  PE  and  QC  samples  should  be  consistent 
with  the  sample  validation  procedure. 

■i 

Similar,  though  not  as  serious,  concerns  exist  for  the  air  monitoring 
data. 

(5)  Table  1 1 1-6  on  Page  II 1-35  of  the  draft  report  presents  relative 
standard  deviations  for  replicate  measurements  of  Method  624  and  Method 
625  analytes  in  laboratory  control  standards  (LCS).  Table  7.2.1,  Pages 
7.2.2  to  7.2.4  of  the  Quality  Assurance  Plan  lists  control  limits  for 
the  analyses  of  laboratory  control  standards.  If  the  analysis  of  the 
laboratory  control  standard  yielded  values  outside  the  control  limits, 
the  laboratory  was  directed  to  check  analytical  procedures,  analyze  a 
second  reference  sample  and,  if  still  out  of  control,  invalidate 
results.  A comparison  of  Table  II 1-6  with  the  control  limits  for  the 
LCS's  indicates  the  performance  for  certain  of  the  laboratories  was  out 
of  control  for  a significant  fraction  of  the  time.  For  example,  the 
control  limit  for  benzene  (item  6 of  Table  7.2.1  on  Page  7.2.2)  was  79 
to  107  percent  recovery,  a range  of  15%  around  the  average  of  93  per- 
cent recovery.  The  relative  standard  deviations  of  the  recovery  of 
benzene  for  two  of  the  six  laboratories  included  in  Table  1 1 1-6  were 
well  in  excess  of  this  ±15%  control  limit.  Similar  statements  can  be 
made  about  other  compounds  listed  in  Table  1 1 1- 6 . EPA  should,  in  its 
report,  explain  this  apparent  discrepancy  in  the  quality  control  -ll 

■■ 

(6)  The  quality  assurance  program  for  the  air  samples  addressed  the  problem 
of  artifactual  contamination  of  TENAX  by  benzene  and  toluene  if  samples 
were  analyzed  more  than  30  days  after  final  cleaning  of  the  TENAX.  Hie 
significance  of  such  contamination  could  be  better  assessed  if 
experimental  results  were  presented  showing  variability  in  background 
levels  of  these  compounds  as  a function  of  time  after  cleaning. 

Finally,  EPA  has  not  used  the  full  range  of  quality  assurance  data  available  j 
to  it  to  justify  its  acceptance  of  data.  The  incomplete  use  of  quality 
assurance  data  is  discussed  in  other  sections  of  this  Appendix.  j 


34 


C.  What  conclusions  has  EPA  drawn  from  the  sample  splits  and  the 

intercomparison  of  data  from  single  sampling  sites  as  to  the  precision 
of  the  data?  Are  such  conclusions  justified?  If  not,  what  conclusions 
could  be  drawn  and/or  what  justification  should  be  presented  to  support 
such  conclusions? 


EPA  has  drawn  few  conclusions  concerning  the  precision  of  the  data  from  the 
sample  splits  and  intercomparison  of  data  from  single  sampling  sites.  This 
is  primarily  a consequence  of  the  small  fraction  of  samples  containing 
concentrations  of  analytes  above  their  detection  limits. 

Triplicate  samples  were  collected  by  the  EPA  contractors  in  the  water 
monitoring  program.  Five  percent  of  the  samples  taken  consisted  of  such 
triplicates.  Seventy-five  percent  of  the  samples  analyzed  with  Method  625 
contained  no  analytes  above  the  quantitation  limit.  This  would  imply  that 
only  25  of  the  triplicate  samples  had  quantifiable  concentrations  of 
analytes  present  (25%  of  5%  of  the  total  number  of  validated  water  samples 
listed  in  Table  1 on  Page  11  of  the  draft  report).  In  the  draft  report,  EPA 
states  that  similar  observations  were  made  with  the  Method  624  and  the 
Method  608  analytes.  EPA  concluded  that  this  nunber  was  too  small  to  give  a 
reliable  estimate  of  method  precision.  The  actual  nunber  of  triplicate 
samples  containing  quantifiable  concentrations  of  analytes  should  be 
reported  by  EPA  along  with  estimates  of  precision  derived  therefrom  and 
these  estimates  compared  with  the  estimates  in  Table  1 1 1-6  on  Page  1 1 1-35 
and  with  additional  estimates  made  from  quality  control  and  performance 
evaluation  samples. 

Duplicate  samples  collected  in  the  air  monitoring  program  were  used  to 
estimate  analytical  precision.  The  precision  was  based  on  the  standard 
deviation  of  the  observed  differences  in  reported  values.  This  procedure 
has  the  desirable  feature  of  taking  into  account  all  aspects  of  the 
monitoring  process.  The  three  compounds  listed  in  Table  V- 5. on  Page  V- 20  of 
the  draft  report  for  which  a large  nunber  of  duplicates  were  obtained  were 
the  three  compounds  identified  in  this  study  to  be  artifacts  on  the  TENAX 
columns:  benzene,  toluene  and  tetrachloroethyl ene.  The  standard  deviations 

appearing  in  Table  V-5  for  these  three  compounds  reflect  both  the 
variability  in  analysis  and  in  the  actual  concentrations  of  the  three 
compounds  as  artifacts  on  TENAX.  Therefore  precision  estimates  deduced  from 
Table  V-5  for  these  compounds  are  not  transferable  to  other  compounds. 

These  difficul ities  in  interpreting  the  significance  of  data  from  duplicate 
samples  have  been  pointed  out  by  EPA  on  Page  V - 12  of  the  draft  report. 

No  data  are  presented  on  those  replicates  which  might  give  insight  into  the 
measurement  process  near  the  limit  of  detection,  i.e.,  for  cases  where  the 
detected  compounds  in  at  least  one  of  the  replicates  are  reported  at  trace 
concentrations.  Comparison  of  replicates  for  which  at  least  one  of  the 
compounds  was  reported  as  below  the  quantitation  limit  should  be  tested  for 
consistency  with  the  limits  of  quantitation  reported  by  EPA. 
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D.  What  are  some  possible  limitations  of  the  quality  assurance  program? 


The  major  difficulty  which  will  be  encountered  in  further  analyses  of  the 
data  acquired  at  Love  Canal  will  be  coping  with  1 aboratory-to-1 aboratory 
variability  in  performance.  No  additional  limitations  in  the  program  have 
been  found  beyond  those  discussed  in  the  answers  to  the  other  questions  of 
this  section.  In  summary,  EPA  has  not  used  all  of  the  quality  assurance 
data  it  has  collected  in  assessing  the  performance  of  the  analytical 
1 aboratories,  i.e.,  tables  of  precision,  accuracy,  and  detection  limits  are 
based  on  limited  sets  of  quality  assurance  data  and  have  not  been 
conf i rmed. 


E.  Were  the  abilities  of  the  participating  laboratories  to  detect  trace 
levels  of  targeted  and  non-targeted  compounds  adequately  tested  by  the 
qual.ity  assurance  program?  What  evidence  does  EPA  present  to  justify 
claims  of  laboratory  performance?  Is  additional  evidence  necessary  to 
support  such  claims  and,  if  so,  what? 


The  quality  assurance  program  did  not  adequately  address  the  question  of 
minimum  detection  limits.  Each  of  the  participating  laboratories  was  asked 
to  estimate  its  limit  of  quantitation  (for  the  water  program)  or  detection 
(for  the  air  program).  These  estimates  have  been  tabulated  in  the  draft 
report.  The  ability  of  the  participating  laboratories  to  detect  trace 
concentrations  (that  is  concentrations  between  the  limit  of  detection  and 
the  limit  of  quantitation)  was  not  formally  evaluated  as  part  of  the  Love 
Canal  Monitoring  Study.  As  stated  in  Section  IV. B above,  the  participating 
laboratories  did  not  appear  to  be  achieving  the  levels  of  detection  with 
Methods  624  and  625  which  had  been  achieved  in  other  studies.  Greater 
attention  should  be  paid  to  the  estimates  of  the  limits  of  detection.  These  j 
estimates  should  be  confirmed  with  other  quality  assurance  data  wherever 
possible.  The  methods  used  to  estimate  and  confirm  the  limits  of  detection 
should  be  clearly  described.  This  point  is  critical  because  all  conclusions 
of  the  draft  report  are  bounded  by  the  limits  of  detection  achieved  in  the 
study. 

The  ability  to  detect  and  identify  non-targeted  compounds  was  evaluated  by 
the  subsequent  GC-MS  audit  (see  Section  VII  below). 


F.  Beyond  the  acceptance  or  rejection  of  data,  what  conclusions  has  EPA 
drawn  from  the  analysis  of  quality  control  samples,  field  blanks,  and 
samples  from  control  areas  as  to  the  precision  and  accuracy  of  the 
reported  data,  the  overall  performance  of  the  participating 
laboratories,  or  the  minimum  detectable  limits  of  targeted  or  non- 
targeted  compounds?  Are  these  conclusions  justified?  If  not,  what 
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conclusions  could  be  drawn  and/or  what  evidence  should  be  presented  to 
justify  such  conclusions? 


EPA  has  used  a portion  of  the  quality  assurance  data  to  estimate  precision, 
accuracy,  and  limits  of  detection  for  the  acquired  Love  Canal  data. 

Estimates  of  these  quantities  were  supplied  by  EPA  or  by  the  contract 
laboratories  with  little  documentation  as  to  how  the  estimates  were  made. 

The  conclusions  of  the  study  may  be  sensitive  to  patterns  and  comparisons 
near  the  limits  of  detection,  and  therefore  the  determination  of  these 
limits  must  be  described  with  greater  clarity  and  the  limits  confirmed. 
Precision  estimates  were  based  on  different  types  of  data  for  air,  water, 
soil  and  sediments,  and  biota  samples.  For  the  air  samples,  the  precision 
estimates  were  based  on  comparisons  of  duplicate  samples  (see  Section  V.C 
above).  For  soil  and  sediments  and  for  water  samples,  the  estimates  of 
precision  were  based  on  the  analysis  of  laboratory  control  standards  (LCS) . 
Not  all  targeted  analytes  were  included  in  the  laboratory  control  standards. 
Table  III-6  on  Page  III- 35  lists  the  relative  standard  deviations  (RSD)  for 
the  recovery  of  compounds  in  the  LCS's.  Some  entries  are  missing, 
suggesting  lack  of  adherence  to  quality  assurance  protocols.  Table  III-6 
would  be  improved  if  it  included  the  number  of  LCS's  for  each  compound  for 
which  relative  standard  deviations  were  calculated.  As  noted  above,  EPA 
should  also  use  the  data  from  quality  control  and  performance  evaluation 
samples  to  confirm  the  claims  of  precision  and  accuracy  appearing  in  the 
draft  report.  These  same  comments  apply  to  the  precision  estimates  for  the 
analyses  of  soil  and  sediments. 

The  accuracy  of  air  methods  was  estimated  from  extensive  studies  of 
calibration  check  samples.  The  accuracy  of  water  methods  was  estimated  from 
the  data  validation  procedures  (discussed  elsewhere  in  this  review)  and  the 
laboratory  reports  on  the  analyses  of  LCS  samples.  EPA  did  not  estimate  the 
accuracy  of  soil,  sediments,  and  biological  data  because  the  relative  matrix 
effects  of  spiked  versus  natural  samples  were  not  known. 

Given  the  imprecision  of  the  analyses  as  tabulated  in  the  draft  report,  it 
is  inappropriate  to  comment  further  on  the  presence  or  absence  of  bias. 

Because  EPA  has  not  fully  utilized  its  quality  assurance  data,  it  cannot 
substantiate  the  claim  on  Page  67  of  the  draft  report: 

"As  a result,  the  Love  Canal  data  are  among  the  most  carefully 
validated  environmental  measurements,  and  (given  the  constraints 
previously  mentioned)  are  representative  of  the  current  state-of-the- 
art  in  environmental  measurement  methodology  in  terms  of  precision, 
accuracy  and  specificity." 
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VI.  Data  Reduction  and  Analysis 

Scope:  To  determine  if  the  data  reduction  and  analysis  are  consistent  with 

and  supported  by  the  quality  of  the  reported  data. 


A.  Has  evidence  been  presented  to  demonstrate  the  accuracy  of 

transcription,  collection  and  tabulation  of  data?  If  not,  what 
additional  supporting  documentation  is  needed? 


The  report  of  the  prime  contractor,  GCA,  presents  compelling  evidence  that 
sufficient  care  was  taken  in  the  transcription  of  the  data  and  in  the 
verification  of  the  accuracy  of  the  transcription  and  transmittal  of  the 
data  into  the  final  data  base.  We  have  no  reason  to  believe  any 
significant  errors  were  introduced  into  the  validated  data  base  through 
errors  in  the  transcription  and  transmittal  of  data. 


B.  Has  EPA  assigned  accuracy  and  precision  limits  to  the  reported  data? 
Have  these  precision  and  accuracy  claims  been  adequately  justified  by 
the  quality  assurance  program  and  the  available  literature  on  the 
monitoring  methods  used?  If  accuracy  and  precision  limits  have  not 
been  assigned,  does  the  documentation  provided  by  EPA  indicate  that 
they  could  be  assigned?  If  so,  what  documentation  should  EPA  provide 
to  justify  precision  and  accuracy  claims? 


EPA  has  not  incorporated  precision  and  accuracy  estimates  into  the 
val idated  data  base. 


EPA  has  used  data  generated  by  the  quality  assurance  program  to  estimate 
precision  and,  in  some  cases,  accuracy  for  the  determination  of  particular 
analytes  in  particular  media.  It  has  been  noted  in  several  sections  of 
this  Review  that  EPA  has  not  used,  as  fully  as  it  could,  the  data  generated 
by  its  quality  assurance  program  to  confirm  its  reported  estimates  of 
precision,  accuracy,  and  limits  of  quantitation.  EPA  has  not  compared  the 
precision  and  accuracy  estimates  derived  from  the  quality  assurance  program 
of  the  Love  Canal  Monitoring  Study  to  estimates  derived  in  other  studies 
and  published  in  the  scientific  literature. 


C.  Has  EPA  adequately  interpreted  the  significance  of  "none  detected"  or 
"below  detection  limit"?  Has  EPA  listed  and  supported  detection  limits 
for  its  targeted  compounds?  Has  it  supported  "below  detection  limit" 
claims  with  documentation  from  its  quality  assurance  program?  Are 
there  ways  in  which  EPA  could  present  the  organic  analytical  data  to 
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make  "below  detection  limit"  more  useful  or  should  it  continue  to 
report  such  data  in  such  a manner? 

EPA  has  provided,  in  its  draft  report,  estimates  of  the  limits  of 
quantitation  and,  in  some  cases,  limits  of  detection  for  particular 
analytes  in  particular  media.  Methods  used  to  arrive  at  these  limits  are 
not  described  and  should  be.  Wherever  possible,  additional  data  from  the 
quality  assurance  program  should  be  used  to  support  the  reported  estimates. 
Additional  comments  on  the  limits  of  detection  and  quantitation  can  be 
found  in  Sections  IV  and  V of  this  Appendix. 

EPA  has  not  discussed  in  its  draft  report  how  these  limits  of  detection 
could  have  affected  its  conclusions,  a central  issue  since  most  of  EPA's 
conclusions  are  based  on  comparisons  of  "detects"  with  "non-detects If 
the  limits  of  detection  are  well  below  the  level  at  which  EPA  would 
recommend  some  alternative  actions  to  those  appearing  in  its 
recommendations,  then  EPA  should  so  state.  If,  on  the  other  hand,  the 
conclusions  would  be  altered  if  more  sensitive  or  more  accurate  data  were 
available,  EPA  must  consider  this  in  its  report.  In  any  case,  the 
conclusions  drawn  in  the  EPA  report  must  include  a statement  of  the 
concentration  levels  at  which  the  conclusions  are  valid. 


D.  What  conclusions  has  EPA  drawn  from  intercomparison  of  sample  splits 
and  sample  campaigns  (the  collection  of  a number  of  samples  from  a 
given  location  over  the  duration  of  the  study)?  Are  the  conclusions 
justified? 


The  use  of  sample  replicates  has  been  addressed  in  Section  V.C  above. 
Because  of  the  small  fraction  of  samples  containing  levels  of  analytes 
above  the  detection  limit,  the  conclusions  drawn  were  limited  and  were 
used  to  estimate  levels  of  precision  for  some  of  the  methods  and  some  of 
the  sampled  media.  EPA  has  not  compared  variability  at  a given  site  over  a 
period  of  time  with  analytical  variability.  Such  comparisons  may  not  prove 
to  be  i 11 uminating,  but  would  be  well  worth  investigating. 


E.  Has  EPA  adequately  justified  the  rejection  of  data  and,  if  not,  should 
some  data  which  have  been  rejected  be  reincluded  in  the  validated  data 
base? 


The  rejection  of  data  has  not  always  been  clearly  explained  (see  Section 
V.B).  For  data  whose  rejection  has  been  clearly  explained,  the  rather  wide 
tolerances  allowed  for  validation  make  it  unlikely  that  acceptable  data 
were  rejected  by  EPA. 


F.  Has  EPA  adequately  justified  the  inclusion  of  data  in  the  validated 

data  base  and,  if  not,  what  kinds  of  analyses  should  EPA  perform  on  the 
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sample  data  and  the  quality  assurance  data  to  provide  such 
justification? 


EPA  has  not  addressed,  in  its  draft  report,  the  probability  of  false  "non- 
detects"  in  its  validated  data  base.  The  GC-MS  audit  has  given  some 
insight  into  the  fraction  of  compounds  missed  or  misidentified  at 
concentrations  near  the  detection  limit.  Such  considerations  become 
important  only  when  the  conclusions  of  the  study  are  likely  to  be  affected. 
The  statements  and  recommendations  made  in  Section  VI. C addressing  the 
significance  of  limits  of  detection  apply  here  as  well. 


G.  Has  EPA  incorporated  precision  and  accuracy  statements  into  its 

validated  data  base?  If  not,  does  it  have  sufficient  information  to  do 
so?  If  it  has,  has  it  been  done  correctly? 


Precision  and  accuracy  statements  have  not  been  incorporated  into  the 
validated  data  base.  Some  precision  and  accuracy  statements  have  been 
given  in  the  appendices  to  the  EPA  draft  report  and  those  wishing  to  use 
the  Love  Canal  data  for  further  study  must  refer  to  these  estimates  when 
interpreting  the  data. 

As  indicated  earlier  in  this  Review,  EPA  should  use  to  the  fullest  extent 
the  available  data  to  confirm  these  estimates  of  precision  and  accuracy. 


H.  How  have  recovery  factors  been  included  in  the  validated  data  file?  Is 
it  clearly  stated  how  this  was  done?  Was  the  procedure  justified? 


Data  in  the  validated  data  base  have  not  been  'corrected'  by  division  by 
recovery  factors,  which  is  consistent  with  current  accepted  practice. 
Recovery  factors  have  been  presented  for  some  of  the  targeted  compounds  for 
the  air  monitoring  program  but  not  the  water  monitoring  program.  Recovery 
factors  should  be  presented  for  all  targeted  compounds  whenever  possible. 
Recovery  factors  for  the  air  monitoring  program  were  calculated  from 
performance  on  calibration  check  samples.  In  a similar  manner,  recovery 
factors  for  the  water  monitoring  program  analytes  should  be  obtainable  from 
laboratory  control  standards,  and  from  performance  evaluation  and  quality 
control  samples.  Recovery  factors  are  invaluable  when  comparing 
environmental  data  from  several  laboratories. 


I.  Should  any  of  the  reduced  data  tables  be  published  as  they  now  stand? 


In  discussions  with  EPA,  following  submission  of  the  study  template  to  EPA, 
it  was  determined  and  mutually  agreed  upon  that  this  question  was  beyond 
the  scope  of  the  review  requested  by  EPA. 
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/II.  The  EPA  Audit  of  the  GC-MS  Computer  Records 


Scope:  To  determine  if  the  EPA  audit  of  the  GC-MS  computer  records 

provides  usable  information  on  the  quality  of  the  organic  analysis  data 
and  to  suggest  ways  in  which  information  obtained  from  this  audit  might  be 
used. 


A.  What  sources  of  error  would  have  and/or  has  such  an  audit  uncovered? 
Does  the  audit  suggest  that  significant  nunbers  of  such  errors  might 
be  present  in  the  validated  data  file?  Could  such  errors  affect 
substantially  any  conclusions  which  might  be  drawn  by  EPA  or  anyone 
using  the  data? 


The  EPA  audit  reviewed  only  the  interpretation  of  GC-MS  computer  records 
(tapes)  and  was  not  a review  of  all  the  steps  of  the  analyses  conducted 
by  the  contract  laboratories.  EPA  reviewed  the  computer  records  for  5%  of 
the  water  and  sediment  samples.  The  two  indices  developed  for  the  audit 
give  a measure  of  the  agreement  between  two  laboratories.  They  do  not 
distinguish  between  mi sidentif ications  and  missed  identifications.  For 
example,  if  only  one  compound  was  present  and  was  detected  by  both 
laboratories,  but  was  identified  incorrectly  by  one  laboratory,  the  same 
indices  would  be  obtained  as  in  the  situation  where  two  compounds  were 
present  but  each  laboratory  correctly  identified  only  one  compound  and 
each  identified  a different  compound.  EPA  gives  no  additional  indication 
of  the  nature  of  the  differences  uncovered  by  the  audit.  For  example, 
were  differences  primarily  in  the  identification  of  two  targeted  compounds 
of  similar  chemical  structure? 

The  use  of  only  three  comparisons  between  EMSC  and  AERL  to  establish  the 
standard  for  agreement  against  which  all  other  agreements  will  be  compared 
does  not  seem  reasonable. 

These  concerns  aside,  the  GC-MS  audit  provided  valuable  insight  into 
discrepancies  in  compound  identification  and  into  the  effectiveness  of  the 
identification  of  non-targeted  compounds. 

The  discussion  on  Pages  VI-11  and  12  concerning  the  concentration  levels 
at  which  disagreenent  occurred  is  convincing.  No  evidence  is  presented, 
however,  to  support  the  claim  "the  great  majority  of  discrepancies 
...[were]...  in  samples  that  were  identified  as  heavily  contaminated." 
Assertions  concerning  whether  missed  identifications  were  primarily  in 
samples  already  identified  as  contaminated  as  opposed  to  samples 
considered  free  of  contamination,  if  false,  could  influence  the 
conclusions  of  the  report.  The  seriousness  of  this  concern  would  also 
depend  on  the  concentration  levels  at  which  EPA  is  basing  its  conclusions 
and  recommendations  as  discussed  in  Section  VI. C above. 

The  EPA  draft  report  cites  as  one  of  the  major  safeguards  in  the 
monitoring  study,  the  identification  in  each  environmental  sample  of  the 
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20  most  abundant  non-targeted  substances.  Eighty  samples  (water,  soil  and 
sediments)  were  audited  for  the  identification  of  non-targeted  substances. 
In  58  of  these  samples,  both  the  audit  1 aboratory  and  the  analytical 
laboratory  identified  no  non-targeted  substances.  In  the  remaining  22 
samples,  the  audit  laboratory  identified  84  non-targeted  substances  while 
the  analytical  laboratories  found  only  1.  This  result  indicated  that,  for 
the  samples  audited,  most  laboratories  did  not  identify  non-targeted 
substances. 

Finally,  the  results  of  the  GC-MS  audit  of  water,  soil,  and  sediment  data 
are  not  applicable  to  other  media.  An  independent  audit  of  air  data  is 
necessary  if  similar  information  is  sought. 


B.  Does  the  audit  indicate  there  are  major  differences  in  the  ways 
the  laboratories  analyzed  their  GC-MS  tapes?  How  would  such 
differences  manifest  themselves?  Was  the  audit  adequate  to  detect 
such  differences?  Should  it  have  been? 


The  audit  showed  that  laboratories  differed  in  the  identification  of 
compounds  near  the  detection  limit.  In  its  analysis  of  the  audit,  EPA 
attributed  these  differences  to  differences  in  computer  algorithms  and 
analysts'  judgements. 

The  differences  would  manifest  themselves  as  mi sident if ications  or 
missed  identifications  by  the  analytical  and/or  audit  laboratories. 

The  audit  appears  to  have  been  adequate  to  detect  these  differences. 


C.  Should  all  of  the  GC-MS  tapes  corresponding  to  the  validated  data 
file  be  reanalyzed  using  a single  program?  Can  we  specify  which 
program?  Should  GC-MS  tapes  corresponding  to  rejected  data  also 
be  reanalyzed?  Could  faults  in  the  analysis  of  the  GC-MS  tapes 
have  led  to  incorrect  rejection  of  data  from  or  inclusion  of  data 
into  the  validated  data  file? 


Variability  introduced  by  use  of  different  programs  is  likely  to  be  less 
important  than  the  variability  introduced  by  the  different  analysts  who 
reviewed  the  data. 

Further  audit  of  the  GC-MS  records,  including  review  by  a team  of  analysts 
using  identical  criteria,  would  appear  to  be  necessary,  so  long 
as  conclusions  drawn  by  EPA  in  its  Love  Canal  Monitoring  Study  are 
sensitive  to  the  presence  of  undetected  contaminants  at  concentrations 
near  the  detection  limit. 
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Faults  in  the  analysis  of  the  GC-MS  tapes  are  not  likely  to  have  led  to 
incorrect  rejection  of  data  from  or  inclusion  of  data  into  the  validated 
data  file  because  the  criteria  for  rejection  were  basically  unrelated  to 
possible  problems  with  compound  identifications. 


43 


APPENDIX  B 


DOCUMENTS  RECEIVED  BY  N8S 


Item 

Doc unen t Title 

Date  Received 
at  NBS 

Bound 

and  Titled 

1. 

Quality  Assurance  Plan  Love  Canal  Study 

18  Aug.  81 

2. 

Quality  Assurance  Plan  Love  Canal  Study 
Appendix  A (Sampling  Procedures) 

18  Aug.  81 

3. 

Quality  Assurance  Plan  Love  Canal  Study 
Appendix  B (Analytical  Procedures) 

18  Aug.  81 

4. 

•Quality  Assurance  Plan  Love  Canal  Study 
Appendix  Q (Subcontractors'  QA  Plans) 

18  Aug.  81 

5. 

Love  Canal  Monitoring  Progran 
GCA  QA/QC  Summary  Report 

28  Jan.  82 

Unbound  and  Titled 

• 

6. 

Environmental  Monitoring  at  Love  Canal,  1980 
(Designated  by  NBS  as  OEM-LC-2) 

18  Aug.  81 

7. 

Data  and  Control  Charts  by  Level 
[Air]  — 'Four  Documents' 

21  Oct.  81 

8. 

Audit  of  Gas  Chromatography/Mass  Spectrometry 
Data  (GC/MS)  Provided  by  Love  Canal  Project 
Analytical  Laboratories 

21  Oct.  81 

9. 

GCA  Report  Volume  I,  Proposed  Work  Plan  Love 
Canal  Monitoring  Project 

21  Oct.  81 

10. 

Additional  EPA  Love  Canal  Materials  for  NBS  Review 
(Correspondence) 

21  Oct.  81 

11. 

Supplemental  Documentation  of  EPA  Love  Canal  Quality 
Assurance  Program  (Correspondence) 

19  Nov.  81  1 

12. 

Revised  Version  of  Audit  of 
Gas  Chromatography/Mass  Spectrometry  Data 
Provided  by  Love  Canal  Project  Analytical 
Laboratories  (Item  8 above) 

I 

19  Nov.  81 

i1 

1 
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13. 

U.S.  EPA  Environmental  Monitoring  at  Love 
Canal  (Draft  Report,  Designated  Copy  #9) 

17  Dec.  81 

14. 

Appendix  I 

List  of  Substances  Monitored  at  Love  Canal 

17  Dec.  81 

15. 

Appendix  II 

Comparative  Data  and  Existing  Standards  for 
Substances  Monitored  at  Love  Canal 

17  Dec.  81 

16. 

Revised  Sections  3.3,  3.3.1,  and  3.3.2  for 
EPA  Draft  Report 

28  Jan.  82 

17. 

Quality  Assurance  for  Water  Samples 
Revised  Appendix  III  for  Draft  Report 

28  Jan.  82 

18. 

Quality  Assurance  for  Air  Samples 
Revised  Appendix  V for  Draft  Report 

28  Jan.  82 

19. 

Report  on  the  Audit  of  Gas  Chromatography/Mass 
Spectrometry  Data  Provided  by  Love  Canal 
Project  Analytical  Laboratories 
New  Appendix  VI  for  Draft  Report 

28  Jan.  82 

20. 

Quality  Assurance  Plan,  Love  Canal  Study, 
Appendix  Q,  Revision  1 

28  Jan.  82 

21. 

Quality  Assurance  for  Soil,  Sediment, 
and  Bi ota  Samples 

Revised  Appendix  IV  for  Draft  Report 

17  Feb.  82 

Unbound 

and  Untitled 

22. 

Laboratory  Audits  - Trip  Reports 

Love  Canal  Study 

(Designated  by  NBS  as  OEM-LC-1) 

18  Aug.  81 

23. 

Love  Canal  Study  Area:  Sampling  Sites 

(Designated  by  NBS  as  OEM-LC-3) 

18  Aug.  81 

24. 

Aggregate  of  Data  from  Various  Sampling  Areas 
(Designated  by  NBS  as  OEM-LC-4) 

18  Aug.  81 

25. 

Aggregate  of  Data  from  Various  Sampling  Areas 
(Designated  by  NBS  as  OEM-LC-5) 

18  Aug.  81 

26. 

Aggregate  of  Data  from  Control,  Canal,  and 

Declaration  Areas 

(Designated  by  NBS  as  OEM-LC-6) 

18  Aug.  81 
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27. 

Aggregate  of  Data  from  Control,  Canal,  and 

Declaration  Areas 

(Designated  by  NBS  as  0EM-LC-7) 

18  Aug. 

28. 

Graphical  Displays  of  Air  Monitoring  Data  as 
a Function  of  Sampling  Date 

6 Oct. 

29. 

Quality  Assurance  Control  Charts  & Tables 
for  Water  Samples 

6 Oct. 

30. 

Quality  Assurance  Data  (Tables)  for  Love  Canal 
Soil,  Sediment,  and  Biota 

21  Oct. 

31. 

Soil  Holding  Times  (Internal  EPA  Correspondence) 

21  Oct. 

32. 

Holding  Time  Study  for  Purgeable  Compounds 
in  Water  (Internal  EPA  Correspondence) 

21  Oct. 

33. 

Report  on  Audit  of  PJB  Laboratories  (Internal 
GCA  Correspondence) 

21  Oct. 

34. 

Report  on  Audit  of  Battel! e and  PEDCO 
Laboratories  (Internal  GCA  Correspondence) 

21  Oct. 

35. 

Collection  of  Public  Comments  Received  by 
EPA  Regarding  Methods  624  and  625  and  EPA 
Response  to  those  Comments 

28  Jan. 

81 

81 

81 

81 

81 

81 

81 

81 

82 

!■ 
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